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RESEARCH LABORATORIES FOR THE ENGINEERING SCIENCES 



Members of the faculty who teach at the undergraduate and graduate levels and a number of 
professional engineers and scientists whose primary activity is research generate and conduct the 
investigations that make up the school's research program. The School of Engineering and Applied Science 
of the University of Virginia believes that research goes hand in hand with teaching. Early in the 
development of its graduate training program, the School recognized that men and women engaged in 
research should be as free as jxissible of the administrative duties involved in sponsored research. In' 1959, 
therefore, the Research Laboratories for the Engineering Sciences (RLES) was established and assigned the 
administrative responsibility for such research within the School. • 

The director of RLES himself a faculty member and researcher maintains familiarity with the 
support reguirenients of the research under way. He is aided by an Academic Advisory Committee, made up 
of a faculty representative from each academic department of the School. ThiH^ommittee serves to inform 
RLES of the needs and perspectives of the research program. 

in addition to administrative support, RLES is charged with providing certain technical a.ssistahce. 
Because it is not practical for each (department to become self sufficient in all phases of the supporting^ 
technology essential to present day research, RLES makes services available through the followmg support 
groups Machine Shop, J nstrumentation. Facilities Services, Publications (incklding photographic facilities), 
''and Computer Terminal Maintenance. ^ 
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A SOURCEBOOK FOR THE DESIGN 
OF A 

. REGIONAL ENVIRONMENTAL LEARNING SYSTEM 

VOLUME 5: 

EVALUATING A REGIONAL ENVIRONMENTAL LEARNING SYSTEM 
A PROGRAM EVALUATION MANUAL 



PREFACE y 

This is one of six Vpl.umes of a report which, collectively, 

is intended to be a Sourcebook for bie Desi'gn of a .Regional 

Environmental Learning System. The report was prepared under 

Contract 300-700-4028 with the Office of Environmental Education 

This six-volume report presumes some background concerni^ 

the concept of a Regional Environmental Learning System, and 

with environmental education as a whole. Considerable relevant 

_ background was supplied in Voluino 9 of the 4th Quarterly Report 

(A Descriptive Ana-lysis of Environmental Edu^qtion)^ and in 

the 5th Quarterly Report (Conceptual Basis' for the Design of ,' 

Regional Environmental Learning Systems), both of which artf 

1 • 

.available from the Office of Environmental Education. ^ ^ 
^' ' Volume 1 contain? an Overview of the Sourcebook, \vith 
short summaries of the^other Volumes. 
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PROGRAM EVAHJATION 

No matter who you are, you can divide evaluation 
situations into those you "hafta" and those you "wantaJ' 
Everything, every program, eve^ activity, gets evaluated 
informally. In addition to that there' are the formal 
evaluation studies . There are some where "the boss" or 
"th^e contract" say you must evaluate. /It is nqt uncommbn 
for the sponsoring agency to say, " Further , funding de- 
pends on what the evaluation shows." Or for your own' 
Board to say, "You need to demoj;istrate the impact of this 
program." Then, one way or another, you "hafta" do an 
evaluation. 



In other ~~S^uations you recognize that you are not 
learning enough frtim the informal evaluating that people are 
doing. You* and Vhe rest of the staff deqde that you 
"wanta" h^ve more, complete, more accurate, possibly more 
credible information .\ 
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You &et up a formal evaluation study. You probably 
have high expectations. And you can anticipate dis- 
appointments. It is something like getting a housecleaner 
for the first time. You expect lots ,of help, and sometimes 
you get it, but it adds to your own work too (what with 
social security papers, sharing personal, problems, getting 
"proper" supplies, etc.) and it won'Kaccomplish some of the 
things you care mcist about. / f 

As to the methods of carrying out a * "hafta"^ and a 
"wanta" evaluation , study, there is not much difference, 
^ut the two approaches feel very different to the pqople- 
getting evaluated. There is a great deal" about program 
evaluation that has to do with people's feelings . With 
mandated evaluation y%u usually feel that you do not have 
control of things. But even with the voluntary evaluations. 




the approach you* use, the persons you get to help, the 
in'struments you choose, and the expectations of your 
audiences combine to persuade you that you do not . have 
much control of things there, eit^her. 

Let's face ijp to tlfis business of disappointments right 
off.' Evaluation studies are often .disappointing . Of course 
they are disappointing sometimes because they reveai how 
much the ,^ogram is in trouble. But more "often they arQ 
disappointing because they do riot reveal enough . about 
•anything. The te'chndlogy of evaluation is riot sufficiently 
developed, so that even a well-run study can be counted on 
to give an accurate accounting of program impact or a good 
indication of sources of trouble. A poorly-run study, arid 
most of them are, is likely to provide as much mis- 
informatioh as good information . There is lots of room for 
disappointment. ' , 



But evaluation studies are often worthwhile. In either 
the "hafta" or "wanta" situation, evaluation tirr^e can be a 
time to learn. You can come to see problems in a new way. 
You do get new information ."^Having done the evaluation, 
or even having required it, makes one politically less vul- 
nerable. But, at evaluation time especially, chances are 
good you will establish closer relationsjiips with other 
people concerned about the longevity and effectiveness of 
the program. 

Whether or not it turns* out to be a good experience 
depends largely on how you choose to go about it. Fancy 
measurements and analyses are not essentia]. V/anting to 
, understand more about the program is essential. And 
selecting an evaluation -plan that matches the resources and 
the interests of people involved is essential. Inhere are lots' 
of alternative designs to choose among (at least according 
to the textbogks) but th'ere .may be only one or two ap- 
proaches that really fit your particular situation. The 
fourth chapter of this inanual is an overview of current 
•program evaluation methods and some ideas about them that 
shouy help program people decide which fits the local 
situation . 



Comprehensive regional environmental learning systems 
with multi-year funding will need the work, of highly skilled 
evaluation specialists. But of course, RElrS staff members 
knd participants do not forfeit all evaluation respor^- 
sibilities. They will *do some evaluatjing on their own and 
should help develop and interpret the specialist's Work. A 
manual like this can assist this cqllaboration . 



THE MEANINGS OF EVALUATION 



Part of the reason there are a. number of evaluation 
approaches} is ^the fact that evaluation means different 
things- in different times and places • No evaluation study 
suits everybody, and' an^ evaluation plan that tries is likely 
^to (all short 'in numerous ways. Against such odds eva> 
:^-^i}^ors usually withdraw, to their, own definitions of eval- 
.^i^uation, per'haps ignoring the fact that^it is a- term having 
>*C0|1i:v.entii^ meanings, ' ^ 

To ..Evaluate ds to jud^e. To evaluate is to arrive 
summarily knowing of merit ^nd shortcoming^^ Ac-- 

coi*ding ..to this simple meaning, the cr4teria and standards 
are siinplte^ straight-forward, and agreed on. But simple 
and widely-agreed-up6n criteria do riot adequately tell the 
successes and failures of an environmental- education pro-- 
gram. * . * - 

With complex, diffuse, and politically sensitive^ pro- 
grams the standards for evaluating are just not possible to 
state • ObjectiveiS'^ can be put iorth, even ratified by various 
groups, but it takes only a few minutes of careful scrutiny 
to see that these 04|j||^ectives do not represent many im- 
.portant hopes , fearS , s^d expectations — the many 
concerns that people have about the programs. 

A danger then is that it will be left to the evaluator to 
say which criteria should be honored. The standards may 
by default become those of the evaluator or the evaluator's 
patrqns. 



Those^ standards are not likely to standards that 
reflect community valuing of the program--more likely to be 
those the evaluator is expert at using. Seldom will the two^ 
be the same. Something should be done. One alternative 
is to work harder to tease out the many criteria the people 
have. And to get those written up. That approach has 
often been trie^j^, sand regularly found wanting^ V\fe have so 
little t^ent for capturing in formal language our aims and 
our willin^gness to accept Various outcomes. 

\' ' , . ^ . 

Another alternative is to recognize that evaluation* can 
be done without formal stat^tftents of objectives, criteria and 
standards. It can be done by r^elying *bn^ the implicit 
feelings of merit people have • V^e can evaluate an en^^ 
vironmental edacation program by confronting people with a 
number of. aspects of the fprogram and recording their 
judgments . ' t.h^^se recordings then constitute the evaluation 
report. , , 



But will "tl^jise people make suitable judgments? Will 
theirs be better than the ^valuator's? Who will pick thqae 
people out?--Op is it even important that an evaluatljtjDn 
stadY result in a ' small number of summary judgA«rita? 

' . ■ • ' . -fe, - 

Bart^ MacDonald ^nd' Ernie House, in separate ♦'Writ- 
ings, have indicated ways of making evaluation more detjio- 
craticy inore fair, less authoritarian. Those - waysj call for 
recognition ,of criteria and standards held not just by 
.patrons but^^throughout the society. They call for a pledge 
to honor the pluralism of the sodetv,^ fcJregoing culmination 
of the evaluative act--leaving that flnal judgmental rejspon- 
sibility to readers and users of the evaluation report. 

The purpose of a program evaluation study then bq^ 
comes one of helping people see the prograrn better, arrive 
at their own decisions as to its faults and accomplish- 
ments--to be, in fact, the evaluators themselves-. 



Instead of integrating program improvement or social 
reform, and instead even of integrating theory-building and 
decision making ipto evaluation, this pluralist view calls for 
keeping evaluation separate. It calls for recognition- of the 
existing and ordinary processes for management and con- 
trol, bad as they may be, and for appealing not only to the 
National processes of people, but to their institutions and 
<:onvictions . 



•According to our way of think'ing, the mechanism most 
suited to this purpose is portrayal . By describing the 
prog;-ani,' by showing it as perceived by those close at 
hand*^~or from special vantage point\ by interpreting* it 
according to various frames of reference, the evaluator 
portrays so that others may evaluate . The evaluator en- 
ables audiences to add this vicarious experience, this 
additi(SMnal experience, ,to. existing experiences and recol- 
lections, to enable them to arrive at their own evaluation, 
and courses of action. 



Portrayal, we believe,, is an essential step in making 
evaluation studies more fair , more democi*atic , and more 
-useful jto^he community i But we hasten to add that in 
emphasizing this point of. view and not other:^ wo are speak- 
ing of our preference, relating to our cxperiei,ice and value 
.cgmmitmonts. Many other evaluators find this ^point of view 
too subjective, too elusive, too difficult to do well. They 

I'i 



prefer to use a predetermined specificatiort of qriteria, to 
invest more heavily in instrument development, and to 
commit themselves to the most rationally defensible author- 
itarian point of view. Many choose to incorporate evaluar 
tion studies into information systems . V/e will try to honor 
these definitions of ,evaluation, too, in this evaluation^ 
manual. * 



THE POLITICAL SIDE OF RELS EVALUATION 

^ All program Evaluation studies have political en- 
tanglements ,/ internally and externally. Everyone expects 
an evaluation of the Vietnam War to be tangled in politics. 
David Halberstam's book , The Best and the Brightest , 
/confirmed it eloquently. Many do not expect the ^valuation 
* of a high school workshop on energy problems to be po- 
liticaL But an evaluator has to work hard to ignore the 
strainings of liberal and conservative advocates. And 
probably should not. 

Ernest House summarized the political involvements, of a 
number ^of educationad programs in a book entitled , 
The P olitics of Educational Innovation. David Cohen and 
Gideon Sjoberg , as well, have valuable writings ' on this 
subject. * 

\ ■ • ■ 

One might conclude that only those thH^gs politically 
sensitive will get evaluated formally. Surely, i( they are 
politically sensitive, that will affect the nature of the eval- 
uation study. 



The school and community work of a RELS will be 
cauftht up in political iss4es. . The originators of RELS and 
their later administrators will face political obstacles and 
U3e political ^processes t\p remain vital. • Their evaluations 
too will be at least as political as scientific. Political con- 
straints may lirniTt some of thc^ evaluator'i work, .but ^:hey 
also serve to i>(^mirid us ' of what is relOyVant . Political de- 
mands can influence studies in a Aum^er of ways: the 
publicity given to the planning, the access to sites, the 
reasons given inform^jints for doirtg th<» study, the wording 
Qf questions, the confidentiality oh data storage, the^ 
value-positions considered in interpreting the data , the 
mode of diksemination of results and go on»' 



In Chapter 4 we will arbitrarily divide all evaluation 
designs into that emphasize program processes and those 
that emphasize program impact ^ Experience tells us that 
usually the ^program sponsors, the bureaucrats, and the 
tax-payers want to know about program impact. More 
often, program staff and participants want to know about 
program management and operations. Not always, of 
course. > jT 

The curiosities of someone mandating ('^Naturally, we 
expect you to evaluate what you are doing,") usually go 
directly to the question of whether or not the program is 
having a gobd impact. The following letter might be con- 
sidered a classic • It is a paraphrased version of one 
actually written by a regional prograiji official, urging that 
the evaluation plan be written to focus on accomplishments 
rather than on inner workings. 



Dear Evaluation Specialist; 

Thank you tor sending a draft of your evaluation proposal. Each of your 
issue* questions merits the attention of evaluat9rs. Two procedural changes 
would make the evaluation proposal easier for me to accept. 

First of all, you should identify what or whOm is to be evaluated. Is it the 
contractors, the volunteers, the contract management, the present training 
policy, etc,?- I am in doubt as to how many separate evaluation instruments 
will be developed, I strongly recommend that there be a standardized eval- 
uation instrument to measure the contractor's performance in all of the pro- 
jects. It Ts my understanding that each of the contractors is to develop his 
own performance criteria. It would be helpful to know the comparability 
among those criteria. 

The second suggestion is that the evaluation begin with a complete statement 
of goals wh ich we will want to measure , A brief exercise in which wa tried 
to restate your issue questions proved to us that they are too ambiguous'to 
determine the specific benefit to be evaluated. 

Since these evaluation data are to guide future decisions about training re- 
quirt^ments for our program, I believe we will have to gather data on the 
impact of our field personnel. An evaluation of how efficiently the contracts 
are being adminlstefred is not enough. The probability of getting Congress to 
accept program outcomes^ cast into tangible, quantifiable dollar values is 
better than a defense of the program in terms of^nlocial ideas and anecdotal 
evidence of good works. Some statistical connection among implementation, 
X\\e program's presence in a community, and measurable events in people's 
lives is required. 

Sincerely, 

Associate Director, Region II 

^ u 
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In Chapter 4 we will examine the reasons why this 
distant program official almost surely continued to be dis- 
appointed with the further planning and later results of the 
evaluation work on that program. (The point we make hece 
is that the political nature of environmental education pro- 
grams will have an influence on what the evaluation studies 
attend to. It v/ill influence many choices the evaluators will 
make. 



These political dipiensions were reflected in the concept 
ot a Regional .Learning System around whjioii our, own work 
in environmental education jcentered ^in 1978-79. We heard 
proponents describing:.' Rj:'J.S as a "new., social form" charac- 
terized by the following conditions : 



(1) Participants in the RELS see environmental issues 
as having major impact on their .region. 

(2) They believe that new approaches to learning and 
management are essential. 

(3) Creative forms of organization — e.g., net- 
working, temporary systems, cooperatives, 
charettes, etc.— (^e needed. " . 

Two illustrations found in this Manuail^re RELS-like 
organizations: an environmental educatioTi— project in" 
Tenrnessee and the Citizens League of Minneapolis-St . Paul. 
The central importance of learning and social interaction is 
featured in case studies in the next chapter. • 

/ •» 

There is dafnger th^t the *sheer demands of organizing 
and maintaining/ a RELS in an atmosphere of tension 'and 
potential conflict are underestimated in the current litera- 
ture, and^p<?rhaps here. For example, there are numerous 
signs th^ citizens rarely think or act in response to "re- 
gions;" and the role of politics as an arena for brokering 
conflicts between opposing sets of values, so amply ill- 
ustrated in the energy /enviornment dilemma, should be 
acknowledged. A description of a Europ.ean scene is ap- 
plicable here too: 

Regional policy has been described as being largely 
an exercise in co-ordinaton; it is a complex effort 
requiring co-ordination of many. . .policies and at 
various levels of government. . . .It should be 
stressed that ip spite of all the efforts make. . . , in 
some cases stretching back for two or more decades, , 
. .regional problems are as acute as thoy have over 
been and current measures appear to be insufficient to 
prevent a continuance of ,such problems, 
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The weight of intellectual tasks iipherent in attempts to 
understand the complexity and scale of environmental pro- 
blems also should be appreciated. Relationships between 
demographic shifts and erstwhile changes in policies con- 
cerning treatment of the environment at all levels of gover- 
nance are both profound and subtle. Their presence may 
be masked by other economic > political and general culturad 
trends. The contemporary population shift to the Sun Belt, 
leaving decline in its wake, ^ and creating problems of 
growth for receiving states and communities, is linked to 
our aging population, to a search for milder climates, to 
retirement policies, and to a ha^ of other less obvious 
factors. I 

It is important to realize (and probably to tolerate) the 
fact that political people call ^for evaluations partly because 
it is politically expedient to do so. Directors appear more 
responsible if they require formal evaluation studies. 
Comtmissipners ward' off criticism and buy time by pointing 
out that an evaluation study is underway. We would nqt 
deny that most calls for evaluations are roOted in a real 
concern for the welfare of the people involved — but a 
self-preservation motive is usually present too. 

Largely for*" this reason millirons of dollars are being 
spent on federally-mandated evaluation studies, many calling 

''for answers to questions that we have not yet learned how 
to answer. The wisest step might be to call a moratorium 
on mandated evaluation studies until we found out how to 

^'do them better, but that will not happen. 

. 

In fhe meantime, some evaluation demands should be 
resisted or ignored. But usually the best response will be 
to do the evaluation — finding a compromise design that tries 
to satisfy the* mandate while providing an inquiry ex- 
perience and some information useful to people most directly 
involved and perhaps to the local community. 

THE ROUTING OF THE READER THROUGH THIS MANUAL 



In a couple of pages this RELS Evaluation Manual will 
plunge into the two case studies: one of an Environmental 
Education Project in southeastern Tennessee, and after that 
a second case study (of the Citizens League in 
Minneapolis-St. 'Paul). The authors will raise questions and 
make suggestions as to modest evaluation studies rhat might 
be done to deal with typical concerns 'in projects like these. 



/ 

/ 

/ 



\ ' ' / . Steps needed to carry out evaluation work will be the 

focus of Chapter 3. In Chapter 4 will be a , plan for a 
formal evaluation unit in a RELS. From there on attention 
wiU be given to major concepts and terms commonly used in 
evaluation work and which should help persons undertaking 
threir first formal evaluation study — to delimit their work" 
an^ to search out. technical help of one kind or another. 

This manual is nbt a cookbook for novice e valuators. It 
is a book for ^ovice evaluato'rs (and for experienced eval- 
uators) and others facing N^ew responsibilities i'n studying 
an environrtiental education program. It is not a cookbook 
because it does not lay out the recipes for a study of 
student gains in achievement or for doing a case study. 
\' / VVhat it does is to set up parallels between the coryt^xt of 

the reader's program and other cantexts (particulaHy one 
in Tennessee and one in i^inn^jj^ Paul) and toHollow 

the routing of thought* that ^tnore experienced evaluation 
specialists go through: mkkihg this/decision, getting that 
authorization; getting into 'these discussions; and so on. 




ERLC 



For more technical matter^s , the reader is directed 
toward other writings. Identification of help awareness is a 
major goal for this brief mai^ah For y examjile , manyTn- 
troductory steps for evaluation studies can be found ih the 
manuals (listed below) prepared at UCLA's Center for the 
Study of Evaluation. * . . ♦ 

. . ■ ■ 

(1) Evaluato^^ Handbook 

^2) How to Deal with Goals and Objectives 

(3) How to Design a Program Evaluation 

(4) How to Measure Program Implementation 

(5) How to Measure Attitudes 

(6) How to Measure Achievement 

(7) How to Calculate Statistics 

(8) How to PreseJtit'*:^ft Evaluation Report 

These manuals are available from Sage Publications Inc., 
275 South Beverly Drive, Beverly Hill, California 90212. 



It ,is apparent that most readers will not read all of 
our manual seqentially, nor even enter through chapter 
one. We think it mdst useful to read from front to back, 
for reasons to be explained in a moment,' but we have tried 
to write it so that readers can route themselves to in- 
tormation most directly useful. For thig reason we have \ 
labored over the section headings and have prepared" the \ 
graphic table inside the front and back, covers. 
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THIS MANUAL'S PEDAGOGXCAL RATIONALE 

0 have ch^en to organize ouV Manual /the> way we did 
partly to emphasize the ^ importance of the local context , the 
importance of organizing the evaluation study in terms of* 
' what ^ happening in the local situations ♦ Most evaluation 
studies reduce the complex phenomena found here and there 
to relationships among a few input and output^ variables. 
Their purpose is largely to combine findings with those of 
Qther sites so the r(B3ults might be generalized and con- 
sidered a general basis for planning; ^* • 

■ ' ' . ^ : ' ' 

We find that kind of generalization not\ sufficiently 
important to justify the loss of attentidn it Vrequires to 
particular conditions and complex interations in the reader's 
own program. We are using our two case studies to keep 
^ those particularistic 'matters as foreground for thinking 
about how to evaluate ^ny reaHife RELS, 



The reader facing a large evaluation responsibility may 
be sinking to the bottom of dismay (a very fcottom-line , so 
to speak) with the chances of rescue seeming remote. Those 
of us writing this Manual have no prescription for miracles, 
and no optimism that new skills of evaluation can be learned 
quickly. 



But old skills may do quite a respectable job. The 
best an evaluato^'r usually can do is what he or she Jias 
been doing already. In this case, it may be taking a 
careful look, lots of careful looks - aided by a number of 
people, at the problems and cpncernis of the program . A 
careful account of tViese can add something "valuable to the 
understanding of the program, particularly if it helps 
readers take a long, deliberate look at the way the' project 
is dealing with environmental issues in this community. 



It seems worth repeating. Whether or not it turns out 
to be a good experience (depends largely on how you choose 
to go about it. FaAcy measurements and analyses are not 
essential. Wanting to^ understand more about the program 
is essential. And selecting an evaluation plan that matches 
the resources and the interests of people involved is essen-^ 
tial. There are lots of alternative designs to choose among, 
but there may be only one or two approaches that really 
fit* The; routing of this Manual is to help the reader tease 
out what questions and listenings may best make it fit. 
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Evaluation takes form and meaning only as it interfaces 
with human activity. Two case studies of environmental 
educatioi? programs are provided in the following . pages. 
These studies are here to serve as a vital context for later 
discussions of evaluation theory and practice. 





ONCE UPON A RIVER: A 9*^SE STUDY OF 
THE LITTLE TENNESSEE VALLEY COOPERATIVE 

Wish that I was on Ole Rocky Top 

Down in the Tennessee Hills k- 

Ain't no smoggy smoke on Rocky Top 
Ain't no telephone bills • 

Rocky Top, you'll alw^s be 
Home sweet, home to me 

Good Ole Rocky Top 

Rocky Top, Tennessee 



'sir 



\ 



Excerpt from Rocky Top . 
'fState song" of Tennessee. 



ERIC 
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Dee'g Cafe, Loudon City 

Nanny, you ready to pay or you 
'just gettin* a paper? 

Nanny opens her handbag. 
Ain't nothing in here. 
How much is it? 

A dollar. ' ■ ^ ^ 

Nanny pays ih change. ' 

Nanny, you ^en \to a doctor? 

No, doctors can^t do 
nothin' ' for me. 

Sometimea^ Could give you something 
for^^^h>*^old, 

^That's what's got me 
messed up ngw. Takin' 
too mutb^^jjij^dicine . Give 
me one of them JDlastic 
bags.* Ain't no™se 
wastin' this good piece 
of meat . 

> 

You need somethin*^ that will knock 
it out of you. I'll take you, if 
you'll go; Which doctor do you go 
to? 

^ I'll go to most any of 
them. Have I paid you? 

Yeah, you done paid us, Nanny. If 
you decide you want to '^o to the 
doctor . . . 




Comiiig 



to know the* Little ' Tennessee Valley Co- 
operative is more than an intellectual probe of' an edu- 
cational .program • It's a knowing of context that seeps 
through your^po^es, beginning from the npoment the Delta 
727 settles down a|t the airport, halfway between Knoxville 
and Maryyille ( pronounced ' Murvul) , and a southern djrawl 
beside you says, "Yeah, feels like we* re^ back on good olf 
Tennessee "dirt • " ^ 



Pm not sure what hearing Rocky Top blasting from, 
jukeboxes and pick-'em--up trucks, hoping Nanny gets to a 
doctor, watching the deer graze in Cade's Cove or redbirds 
play about in the brush beside the Little Tennessee have to 
do v/jth LTVEC; but somehow, ' for this observer, they all 
became one. These experiences were much part of a 
growing attunement to the woven consistency of^ people, 
land, and rfvers; -where change is so slow as to be almost 
im percept able ; \^here the passing of springtime is charted- 
in morning hoar-frosts on the mountains,^ and the' coming of 
dogwood winter; and where environmental education is 
sometimes teaching kids how to clean fish. 
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THE SEVEN SCHOOL DISTRICTS 



The Little Tennessee Valley Educational , Cooperative 
" has , from the start , been made up of seven participating 
school districts — tbe three county districts of the Little 
Tennessee Valley — Loudon, Blount, and Monroe — as well as 
four dty systems within those counties — Lenoir City, Sweet- 
watejr, Maryville, and Alcoa. Altogether, there are ap- 
proximately 29,000 public , school children in the 
seven-county area and within the LTVEC region. By 
regions, the size of the districts, according to student 
population, varies as follows: 



Blount County 11,000 

Monroe County * 4 , 000 
Loudon County 3,000 

Maryville City '2,500 
Lenoir City 2,500 
Alcoa City ^ 1,30"0 
Sweetwater City 1,300 ^ 



There are several distinctions between the county and 
the dty systems. The three county' systems are larger, 
more rural and administrated by elected superintendents, 
whereas the smaller dty systems are adminijstered by super- 
intendents appointed by/their school boards. 

\ 

In addition, each district has a distinct personality 
witli significant geographic, economic, and cultured * dis- 
tinctions. The scope of this study did'^n^ot indude ^ in-depth 
inquiry into each school district. As observations were 
being conducted through the ' districts, however, a few 
quotations and vignettes which tell something of the charac- 
ter of the districts were saved , and are simply shared 
below in the form in which they came to the observer: 

V 

Blount County 
* Blount' is one of the oldest counties in T^nessee, 
having been developed and given official status 
by the Tennessee Territorial Legislature on July 
1 1 , 1795 . The territory forming the new . county 
^ had' originally been part of Knox County. A 

county court for the new county was organized in 
Septem^ber, 1795. ^ - ^ 

Blount County was named for William"' Blount, 
Governor of the Southwest Territory. ^Maryville, 
the county seat, was named in honor of Governor 
Blount^s wife, Mary Graing^ Blount. 



The settlement of the Blount County area got 
way in a substantial way around 1785, when there was 
a* heavy immigratfon from North Carolina and Virginia. 
There were some scattered settlers prior to 178 5. 
Apparently, the earliest white sellers had much 
trouble with Indian raiders* 

* * 

"Blount County is wealthy, but has a low tax rate* Its 
financial base is Alcoa Aljurninum and Knoxville. ,It has 
a very low property tax» People who commute to 
Kno3f:ville don*t care about the political scene in Blount 
County, 'Landed farmers have ^the . political power. 
But they are extremely conservative and don't want to 
pay ' taxes. People in Blount County who are 
well-educated are demanding better ser^rices, but thq 
county can*t afford them, Blount County is eighth or 
ninth in wealth in the state and eighteenth tn .taxes 
collected , " ' 



"Local governments aren'.t going to raise taxes for 
facilities , " 

"C6nsolidation is-* one of our big issues," 



Loudon Count y ^^^^^^^^^,; v 

Loudon County 'became a legal entity on^May 21, 1870, 
and Christiana County, . The name was changed a few 
days later to Loudon County, The county, formed 
from parts of Blount, Monroe, and Roane Counties, 
was named in commenoration of Fort, Loudon, Colonial 
British fort near where the Tellico River flows into the 
Little Tennessee RiveV near Niles Ferry Bridge on 
Highway 411, The fort itself was named for the Earl 
of Loudon, Commander-in-Chief of the British Forces 
in America and also Governor of Virginia , The fort 
was erected in 1756 ; it is , supposed^^"i)ave been the 
first structure of its kind creAtea in Tennessee by 
Anglo-Americans, The fort was destroyed by the 
Cherokee Indians in 1760. 

The first church in the county was built by the ^ev, 
Isaac Anderson, a teacher of Sam Houston and founder 
of Maryville College, The date was 1823 ♦ 

"In our district there were problems around the mid 
'70s — the haftdicapped law and trying ^ to comply, , , , 
Now its not our major problem. In fact, I can't think 
of any major problem now. 



Monroe County 

Monroe County became a legal entity in 1819, having 
been d/veloped from lands obtained by the Hiwassee 
Purchase made by the Federal Government, from th^ 
Cherokee Indians. The county was named for the 
fifth President of the United States, James Monroe 
(1817-1825), a native of Virginia. The present county 
seat was named for President James Madison, the 
fourth President ( 1809--1817) , also a native of Virginia. 
The site for MadisonviUe wa's probably selected in 
1822, but the town >,was not plotted until 1827. Ap- 
parently the first anj^ second county courts were held 
at Morganton (now in Loudon County) and at 
Henderson, east -of MadisonviUe.. The territory, noW 
in Monjroe County, contains the territory occupied by 
*the Cherokee Indian towns of Chot^, Tellico,' Citico, 
and Toqua . The earliest permanent settlers came 
largely from Virginia, South Carolina and North 
Carolina. 



"We^re basically slow to change. The longer I live, 
the more I see me^ to this." 

"Of the seven Co-op ' districts, Monroe County h^s "t^e 
lowest per capdta income. thet tax « base is strickly . 
property. It*s an agricultural-based county. Our" 
biggest challenge is getting {people to agcept respon-^ 
sibilityYof where they are education\yise. Many of our 
children have not had **the cultural' aciyantages of 
things in the home, etc." < • * 

"Our people are a mixture of Appalachian mountain 
people, and small town people. The mountain people 
are slow to accept, defensive, and hold back. They 
are; sUhv/ for , change and very cautious.". 

Lenbir City 

"Most people wark in, Lenoir City in Oakridge, 
Knoxville, or for TVA. This is the place where they 
sleep at -night. Twenty-five perqep.t ' of the kids^ 
^parents are federally connected. I fee^ we are in the 
process of change — industrial development in the city 
and county, a change process , so that the biggest 
concern now is providing for 
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living in this, area^ There is an increase^ of kids 
in the schools. If Eixpo '80 comes to Knoxville, 
and if ^ the Tellico Project ever gets out of the 
county and gets development going ... But we 
had better do some planning. We could not 
accomodate it. " ' ^ •* 

"We feel we never did lea^e the 'basics to go back 
to.. We never did go to the new math. ^ We're 
still somewhat traditional, and if that is. not good, 
I guess we feel It is tri6d and proved." 



ALCOA . * , 

"We have a sig^nificant number of black 
kids — twenty-eight to thirty ' percent . We have 
the. best tax base in the state — from all the plants 
)of the Aluminium Company of. Amer^ CaLCOA) 
that are here." 



. . . «• 

".V/e'^vrf got all we need: we're the richest school 
system in the state. "> ^ 



"Population is a problem— declining enrollment. 
Alcoa retirees are buying > up the houses. It'§ 
difficult to maintain quality." ' 



Sweetwater . 

"There was a settler who flSime out and built a 
house on the hill above the spring. In this 
house, the man had a large molasses barrel.' 
This w^s originally the home 6f the Cherokee 
Indians. One day some Cherokees came down 
through the Cherokee forest, got jdrunk, and 
they went up and se't the man's house on fire. 
The molasses barr«4 tumbeled into the creek, and 
th« Cherokees named the spot 'Spring of Sweet 
Water.' " 

"The people here are of, Scotch-Irish de- 
scent—hardworking, energetic,- outgoing, have 
(fione/a lot of things for themselves." 

"'Basic? are our main cohcern, meeting standards. 
Wg are a traditional school, textbook-oriented. 
We're trying to eliminate things " from the outside 
that might distract from the basics." 

■ 2 1 
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BEGINNINGS • . ' ■ 

The histbry of the Little Tennessee Valley Cooperative 
is not a single 3tory, but several, each a view from a 
different window* Joe Sherlan, superintendent of Sweet- 
water City Schools, was . there and remembers thfe begin- 
nings this way: \ , 

• • ^ ' 

TV A,- in planning for Tellico, decided they 
needed grassroot support. And if Tellicb came to 
fruition they would need educational support . 

^ . . They would have to ckoss county l\nes be-* 

cause the project was across three counties. So 
out of this came the Charrette, , . .and then t;he 

, ' Cooperative ('J.TVEC)v. 

« 

A report by a study group from the University of 
Tennessee describing the activities of the Little Tennessee 
Valley Charette also described the Cooperative's beginnings: 

For about four years the Tell ico Area Planning 
Coandl has been develof^ing plan# for the pro- 
posed "Timberlake" community to encompass parts 
of three counties, Blount, Loudon, and Monroe. 
As otie aspect of this process , the seven super- 
intendents in the area formed a committee to 
consider the educational dimensions of the pro- 
jected community. During their discussions it 
became apparent that it was inadequate to con- 
sider educational planning just for the Timberlake 
community. It was decided that educational 
planning should include the entire three county 
area. 



A group representing the seven public school 
systenjp and the local governments in the Blount, ^ 
Loudon, .and Monroe tri-county area, therefore, 
instigated the development of a series of meetings 
designed to gain widespread community involve- 
ment in education. The "chartette" technique was \ 
selected in order to facilitate multi-group involve- 
ment in the geographical area and to* provide a 
means foi^ studying and resolving educational 
problems within the context of ^otal community 
needs. 

^ ***** 

It is proposed^ that the school systems involved in^ 

the Little Tennessee Valley CHarrette form a 

confederation of local school districts to be called 
the "Little Tennessee Valley Educational Cooperative." 
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It IS further proposed that this educational cooperative be 
formed , under the General Act of the General Assembly of 
the State of Tennessee known as the "Educational 
Cooperation Act," Public Chapter No. 511. This act permits 
"Boards of Education the most efficient use of their powers 
by enabling them* to cooperate ^th other localities on a 
basis of mutual advantage and to thereby provide 
educational services and facilities in a manner that will 
accord best with geographic, economic, population, and 
other factors influencing the needs and the development of 
local educational facilities and services." 

As part of its planning toward the Tellico Dam Projfect, 
to be implemented in the Li)ttle Tennessee River Valley, 
TV A stimulated the concurrent planning of the model dty, 
Timberlake. To parallel these activities, TVA was further 
instrumental in the organization of the TelHco Area Planning 
Council and the eventual organization of the Little 
Tennessee V^illey Educational Cooperative. Tl\e organization 
of the Co-op wa? unboubtedly seen as an advantage by 
TVA. Support of the project and cooperative planning 
among local school people for the future of Timberlake could 
be a foothold in gaining more general support of local 
people for the Tellico Project. Thus LTVEC was created, 
in one sen^, a forced marriage among geographically linked 
but historically independent schqoh districts. IVhy would 
TVA work through the educational systems in gaining 
political support? Because in the Little Tennessee Valley, 
schools were not only central to community life and com- 
munity interest; but also significant in community political 
activity. Or, as one person interviewed put it: 

Education in the South isLJik^ -motherhood and 
^ apple pie. It's looked upon as being sacred . » . 

They can wave the flag more for education than 
for anything. 

For the local districts, the Cooperative was certainly 
not a pro-active, spontaneous coalition of seven school 
districts commirted to cooperative educational ^a^rtivities . 
LTVEC wae instead a local shared reaction, probably more 
political and econofnic than educational, in response to the 
external press of a special environmental circumstance; 
i.e., .Timberlake. These earliest participants in the 
Charrette recognized the wisdom of Unit^ [n the face of the 
coming of Timberlake, an fntruder at once dazzling and 
ominous . 

stakes were hifeh. The location of control of the 
educational system of Timberlake would be a critical problem 
ot the community itsol'f and to surrouncUTTg counties. 

2j 
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It , is imperative that the three counties cooperate 
regarding the Timberlake project. Should the 
project be developed as proposed, Loudon County 
Schoqls would be swamped with studeWs, but 
would have an insufficient tax base to cope with 
the j^roblem, Blount County would have the same 
\ problem, although with less severity. Monroe 

County would contain the majority of the industry 
and the resulting high tax base. An attempt to 
operate the school systems under their present 
conditions would create educational chaos. How 
ever, Monroe County should not be expected to 
share revenue unless the system receives .ade- 
quate services. The need ..for an equitable 
sharing of services and rev^nu^is obvious. The 
fallowing alternatives are offered: • . . (From a 
University of Tennessee report of. the Little 
Terlessee Valley Charrette) 

Additionally, the fin^^^idal burden and/or benefits to 
vbe felt from the development of the model city were of great 
concern arid importance to the seven' school districts. 

Stated purposes and intended activities for the Co- 
operative at its start-up reflect an extension of conflicting 
interests. Interests of local school districts and those of 
outside interests were widely divergent. ^ Those with 'vested 
interests in the Tellico Project conceived the Co-op as 
functioning ^privately within the purposes of planning and 
developme^ of the Timberlake educational system. 
f *^ 

This Utopian idea goes back p6 the late 1960s 
when the Tennessee Valley Authority heeded 
residents' pleas to build a dam on the Little 
Tennessee River at Lenoir City and otherwise 
^ help the sagging economy of the three counties. 

TV A didn4 just want to build, a dam for the sake 
of building a dam. It also conceived Timberlake, 
y named for the British explorer who came to these 
I parts in the 1700s. The water^oriented com- 
munity projected for 50,000 residents would be 
built to spare citizens from pollution and eyesores 
from the beginning. 

New industry brought in would be located else- 
where. This* calls for -new vehicle arteries, in-- 
eluding new bridges over the Little T (one al^, 
ready has been built) , and a school system to 
take care of today' s and tomorrow's children • 
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So the Little Tennessee Valley Educational Co-op (LTVEC) 
was born, supported by joint grants from TV A and the 
AppalacJtxiart Regional Commission. ' (From News Sentinel , 
date ynknown.) ' ' 



Certainly, those within the local districts had the 
need-s of their own existing schools, school populations, and 
programs in mind. The Charrette collected data across the 
three counties to' determine within-county and across-county 
needs to which LTVEC might directs its efforts. 

Such a diversity of programs, to be initiated 
within "^even school districts, undoubtedly will 
require j^ome careful planning and organizing to 
provide (riaximum use of the existing facilities, 
funding, and involvement of resources outside of 
formal education. Throughout the Charrette a 
strong desire for cooperation in attempting to 
solve the educational problems of the Little 
Tennessee Valley was evident among the citizens, 
students, and educators alike • organizational 
refinement is recommended whiclt might be desir- 
able to improve, the cooperative involvement of the 
. ^ seven school systems. ( Fi'om University of 

Tennessee Charrette Report.) ' ^ 

It is interesting to note, also, that in tKe Charrette 
study the individual counties each reported Somewhat dif- 
ferent, specialized, and considerably focused needs, which 
they hoped the co-qp might help meet: Blount County, 
reorganization of high schools, vocational training and 
sources of funding; Monroe County, new facilities and 
vocational training; Loudon County, better facility utili- 
zation, new educational programs, new sources of funding. 

Still, » thfe formal report of the Charrette study was 
concluded with a broad and inclusive statement of purpose 
for LTVEC which would parallel that of a typical educational 
cooperative: 

The primary purpose of the Cooperative shoul/d be 
' to provide specialized educational services on a 
regional basis so that a high degree of equality in 
educational opportunity can be achieved and 
advanced educational practices can be introduced 
and sustained. The Cooperative should perform 
those specialized services which school systems 
are not able to perform efficiently themselves or 
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those which could be performed through a pooling of human 
and material resources. (From University of Tennessee 
Charrette Report.) 

Some of the functions that should be included in the Co- 
operative would be planning', research, development, eval- 
uation, and services through commonly operated programs 
of inservice education, vocational education, special educa- 
tion, supplementary media and materials, early childhood 
education, driver education, educational television, shared 
teachers and consultants,, the use of technological in- 
novations for delivery of programs, and so on. The 
Tennessee State Department of Education and colleges and 
universities should participate with the local, school systems 
in responsible role's in the planning and execution of these 
functions. (From University of Tennessee Charrette Re- 
port . ) 



BUCK ROGERS LAND 



Dr. Oakes called it Buck Rogers- Land. It was 
such an ideal situation: It was real exciting, you 
know. First of all, there would be a completed 
educational center for all-aged people . . li- 
brary, swimming pool, golf c;«urse . , .' idealistic 
. . . classrooms for all nature of the handicapped 
. . . a complete educational park for the three 
counties. ((^uote from Glyden Calhoun, Co-op 
Bookkeeper, I97I to present.) 1 



At the recommendation of the Charrette, a ten-member 
Cooperative Board was formed for LTVEC, with one member 
elected from each of the seven local school boards (Monroe 
County, Loudon County, Blount County, Sweetwater City, 
Lenoir City, Alcoa, and Maryville) and one member from 
each of the three county courts within the Cooperative. In 
August of 197 1, this Board appointed Dr. William Oakes as 
the LTVEC Executive Director, and the Cooperative began 
its operations within one year grants from TV A and ARC 
(■Appalachian Regional Commission). Much of the story of 
the first year or so of operations of LTVEC is related to 
the character and activities of its new director: 



"Bill- was a highly verbal person who' had wide 
contacts and went all directions at once. He was 
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an Idea man ... He was a pretty good fjnancial person, 
fairly well able to hold the Co-op together through personal 
influence . " ' ' 

"He was local, from Knoxville . . .friendly, a 
wheeler-dealer. He traveled to Washington a lot, and made, 
speeches." 



"He was brilliant, committed, but controversial; he wanted 
to expedite thing^g^ didn't alwayjs explain . . . 'had dif- 
ficulty explaining, iSce the budget."- 

"Bill did not see hirtself as facilitating local interactions, 
but interacting with' the Department of I^abor, the press, 
TV A, and so on. 



This first year of the Co-op's operation was intended 
and funded as a planning and development period. Spirits 
were high, ideas inspired if at times grandiose, ar>d news 
releases ' flowed. ^ Most of the Talannin^ at this time was 
related not to planning of cooperative programs^for .the 
seven existing LT^/EC school districts, but to the planning 
- and development of the proposed Human* Resources Center 
to be located at Timberlake, still a dream dty. 



ir 



MADISONVILLE, TN. —There's a sitting-on- 
the-edge-of-the-chair excitement among educators., 
public officials, dvic leadet-s \nd school patrons 
of this region as they speak of a significant new 
concept which id being put into operation to meet 
the. educational needs of a^^^ree-county area. 

A Human Resource F^ark, opening in 1975, with 
structures to house a high school, a middle 
school , and vocational-technical facilities , will 
involve aw initial investment of some $5.2 million 
for cqnstruction on a 300-acre, 5ite northwest pf 
here. c ^ . 

The facilities, to serve the residents of Monroe, 
Blount and Loudon counties, would provide educa- 
tional training opportunities for all ages. 

The program envisions a steady expansion, with 
additional facilities being construct ed<;^ over a 
period of some 25 years, with the total investment 
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in buildings, equipment and grounds reaching $32.8 million 
by some time in the 1^90s. • ' 

The Human Resource Park could-^but would not 
necessarily — take the place of present high sc;^ools operated 
by the seven ^ra^ooi system (some of them municipal) in the 
three counties By Mouzon Peters, Times Tristate Editor.) 

Basic educational needs of tri-counties are already ^ 
apparent, and some possible solutions can be seen* Among 
these is a perimeter comprehensive high school which could 
be part of a Human Resource Center (HRC) which would, be 
somewhere near the junction of the thoee counties, and it is 
possible that industry and government could become in- 
terested and that HRC would develop as a tji-county cul- 
tural center. It could include an Area Technical-Vocational 
School, an Environmental Educational and Research Center 
as well as many other facilities and c services to serve the 
young and not-so-young. (Charrette Forum Release 2/71 . ) 

The Little Tennessee Valley Educational Cooperative 
(Charrette) is continually searching for alternative means to 
solve some of the educational and 'human needs in the area. 
Of much more' modest scope were program efforts, with a 
few meagre starts especially in vocational training, adult 
education, and driver edu^cation. ' 

Later, in retrosp'ect , Bill Oakes summed up the Co- 
operative's first year: 

"A few programs were star^ted ' with federal funds, 
but most of' the year was sp^nt , pursuing the 
fi golden , fleece bf the future.^ In September, 1972, 
bhe fleece began to tarnish and flee further into 
the future." 



4 

LOST DREAMS, BROKEN PROMISES 



At this point, the identify of LTVEC still rested 
heavily upon continuing successful development of Timber- 
lake which, in turn, depended on the successful completion 
of the Tellico Dam (though a few were saying that Timber- 
lake was a ^^ood idea with or without the new reservoir) . 
Boeing Corporation had been contracted to spearhead the 
planning of the residential and industrial components of the 
model city, and paralleling the Co-op's first year of opera- 
tion, Boeing had correspondingly invited $250,000 in their 



part of the overall TimberlaHe effort. MeaLwhile , public 
sentiment and other antagonistic forces were^gpaining sup- 
port in their fight against the construction- of the Tellico 
Dam. TVA, therefore, had become extremely reluctant to 
follow through with its early promises of ^pport, especially 
economic support to the Timberlakd project or to the Co- 
operative. 



Boeing finally withdrew entirely, leaving th^ Timber- 
lake Project essentially without direction and support, and 
leaving LTVEC in a seemingly helpless situation, as well; 
essentially without support, identity, and--most im- 
portant — without a dream or a "cause. The • situation is 
summarized in the following LTVEC Board Meeting Notes: 

Evaluations of LTVEC and " TVA— It wa?*- pointed 
• out to J:he board that there seems a distinct 
possibility that Timberlake will not develop at the 
previously envisioned i^te, and it may be 1980 
. before significant numbers of Timberlake residents 
are on haad. Since the needs of the tri-county 
area have existed for some time, it is obvious a 
'delay of another eight years or so would not be 
appropriate. The Board then discussed the 
possibility ;of returning ' to the original rfecom-^ 
mendations of the Charrette and exploring the 
possibilities of a perimeter high school,' a Special" 
Education facility, and -/a state area voc-tech 
school. 



As these minutes reflect, LTVEC realized 'its only- 
option for survival was <o recoup forces, focus inwardly,.- 
and find within its own ranks of seven school districts the 
individual'-and cooperative • needs which would provide their 
m.utual commitment with a real purpose and an action 
agenda. 

Paradoxically, perhaps,, the withdrawal of external 
support from TVA and Boeing provided Co-op membership 
with the start of a new identity, of a sort, or perhaps the 
return to an old identity that was fclfere all along , ' before 
Timberlake ahd all the rest, not formalized as the Little 
Tennessee Valley Cooperative, but there as a loose- 
ly-coupled network of local people with local concerns, who 
stood sometimes apart and sometimes together for things 
which they cared about--like their children, their families, 
their property, their independence, their simpler life style, 
their right to being slcfw to commit but long in committment. 
When the odds were against them, the rugged individualism 
for which their southern and mountain ancestors have so 
long been^known stood the Co-op well in turning the corner 
sW-'^^^ ^^^^ Bo&rd member has 
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'^Then the th/ust went inward. It was Oakels. 
Plus the fact that there^s a stubbornness in the 
Tennessee personality that says , ^ By Goerge , 
we're gonna make it work whether it's with TV A 
or not.' " 

This transition was ' not to be an easy one, however, 
and found the cooperative entering into a long and painful 
struggle toward redirection, autonomy and self-suffipiency . 
As the last of the TVA and ARC grajjit monies were coming 
available, a new ARC grant in the amount of $65,000 was 
negotiated. The ^new grant was months'! in gaining the 
go-ahead from ARC central offices, leaving LTVEC in in- 
creasingly extenuating financial circumstances. Board notes 
of November .15,^ 1972, reflect the .worsening situation: 

Tteiere apparently comes a timej in the life of each 
^ cooperative when the lag between expenditures 
an^ receipt of a federal check forces the co- 
ot/erative movement in most states, unless they 
are directly supported by the State Department of 
^Education. Our present cash balance is zero. 
The first payment on our TVA contract will come 
after the first of ♦the month, and we still have 
$3,900.00 coming from last year's ARC contract. 
In £he meantime we need> a bank loan. 



Rapidly depleting funds and high overhead costs led ^ 
the co-op 'in the spring oj 4973 to move from its location in 

^ Greenback to empty s^Tace in ALCOA School District. 
When the ARC grant monies finally came available, the total 
amount was reduced substantially from the original budget, 
due to an agency policy which disallowed retroactive reim^ 
bursements. Salary checks were delayed for weeks, the 
Co-op found itself burdened with a debt it had no re- 

' sources to repay and morale hit a new low. Glynden 
Calhoun described the final page of this rather dismal 
chapter in the life of the Co-op as follows:. 

In '73, when we 'moved up to ALCOA, there was 
not much mox% activity. We couldn't keep^oing' 
on promUes. We had to have money. ... So 
long as Tlmberlake was being planned, there 'was 
enthusiasm. There was TVA with all their money. 
And Boeing. And ARC . ^ (But there were] 
growing pains. You can't sit still. I've heard 
Dr. Oaks say, "You have to grow or give up." 

For more than a year, the Co-op was engaged in an 
active struggle for survival, an almost day-to-day battle to 
• create a new Identify, to set new and realistic goals, 'and to 
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become financially solvent • As a Strong resentment de-- 
veloped . against the fickleness of outside force.s and 
agencies; a^ considerable rporal investment was made by Bill 
Oakes and Board members in^'somehow finding ways to 
become self-sufficient, . - 



During this period^- Board membership ^ previously 
including, only school board and county court members, was 
enlarged to include superintendents from the seven school 
districts . This change modified the governanace ^ind un- 
doubtedly the vision of the Board from one of more general 
political interests (and perhaps somewhat limited com- 
mittment) to greater investment in sqhool-related issues* 

The Co-op had maintained a poor record, to this date, 
in development of effective new pi-ograms. It had placed 
minimal investment of resources into a* smorgasbord of 
programs for the district, none of which se.emed particularly 
vital or of high priority to anyone* -> 



Bill Oakes realized that the Co-op needed a ney cen- 
/ tral and motivating theme which could somehow generate 
both enthusiasm and funds for Co-op operations* A first 
opportunity of this sort occurred with the Co-op's attempts 
to develop a viable vocational educational program. Voc Ed 
seemed a likely lifeboat; two years befor^pivthe Charrette 
had listed it as a top priority need across all three 
counties • Even more important now, the state was cur- 
rently making plans for support of planning and develop- 
ment of new vocational education centers across the state • 
The hope was that LTVEC might assume administrative 
responsibility for a cooperative Voc Ed program effort 
among th6 seven school districts* The outcome of state Voc 
Tech plans, however, was that the staxe finally provided 
^ the physical structures but not monetary support for pro- 
gram development, so, in the end, the various district^ de- 
veloped their own ^oc Tech programs without LTVEC, 



Concurrently, the emergence of 'an all-imp^yrtant/ jffro- 
gram, psychological services, had later begun to emerge as 
a program to hold a central^ place over several years in the 
purpose, vitality, and services of LTVEC. Paradoxically, it 
seems, the full potential of the program was not at first 
clearly perceived, as can be seen in the. following rather 
perfunctory summary appearing in . "Cooperatively 
Speaking," an LXVEC newsletter: * 



LTVEC* s last bucket of money from the federal 
well has been approved, and, with this, programs 
for the school year 1973-74 have been planned 
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and submitted to the respective Boards of Edu- 
/ cation. A serious problem for years has been the 
lack of being able, to test students to determine 
•the programs they need. Such a program is 
called psychologrcal services, and LTVEC has 
proposed to provide the service. 

A school psychologist has been found, and, with 
interns from UT and help 1from the "State De- 
irtment of Education, screening of students is 
poposed to begin jia September, 1873. 



BETTER YEARS 



With 1973 dawned the beginning of a new era for thfe 
Co-op. That year, Tennessee Public Law 839 came into 
effect along with additional state funding to hasten com- 
pliance. The law established as state policy to require 
school districts to. provide special education services suf- 
ficient to meet the needs and maximize the capabilities of 
handicapped children. 



This new 'mandate loomed large and threatening for 
most school districts. First, the state did not interpret the 
law, nor did it prescribe^ the logistics of^ how compliance 
might occur. Secondly, the educational implications of the 
law ran contrary to the existing fabric of schooling being 
practiced by many districts. As one observer put it, "The 
law was antithetical to the educational system in the 
state — a bureaucracy in which educators run the show." 



The seven LTVEC districts were sufficiently alarmed 
about the prospects of the new legislation for educating the 
han^capped to agree to the cooperative pooling of re- 
sources in order that the Co-op might develop* a full-scale 
psychological service programs, complete with an added 
staff person to see to its sXiccess. Subsequently, this 
person, . Dr, Jerry Morton, was hired. He immediately 
embarked upon , a mission to successfully utilizetjae^andi^ 
capped Education Law, with its ^rir^iirnKyanoo^«^i'^<?rr^' K^^^i^ifo 
as a vehicle for pulling the Coopera^tive together, streng-» 
thening pt-ograms, enlarging staff, balancing the budget, 
and shaping a new dream for the Co-op ^ a dream which was 
to include words like "individualized insti^uction," "advocacy 
for children , " " development^ of human resources , " and 
"educational revolution." 
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The new Psychological Services program included two 
major thrusts.- First, a cadre of University of Tennessee 
graduate students in psychology were hired on a part-time 
basis to go about doing outreacl;! wqrk in testing and place- 
ment for the school districts. This group wfot only achieved 
delivery of services to schools but theyalso quadrupled the 
Co-op stStff and generally infiltrated the ranks with youth- 
ful idealism and prioAtie? which placed concern for handi- 
capped children above concerns for agency, di§trid:s, 
PQ^^eVf politics, and economics. ' / 

St 

■ . c 

In the second thrust t)f the new psychological services 
program, Jerry Morton began to work personally with ^ the 
Co-op districts in arriving at viable interpretation of Law 
839, in providing badly-needed legal assistance and ccyi- 
sultation to the districts themselves. 



The perceived utility to districts otf the Psychological 
Services Program over the past six years is readily doc- 
umented across the several districts: 



Monroe County ; 

About SIX years ago we began our program of 
individualized instruction/ We'r^ about halfway 
there; the Handicapped Program has helped a lot. 
Our whole intent is not for the accelerated but 
for the lagging child. We have been guilty^ of 
teaching to. the mass. Public Law 839 really puts 
focus on the lagging child. 



y ' Blount County : 

Tt [ assistance from the Co-op] was very nacesir . 
sary at the beginning. We wouldn^t have. iDeen 
able to do the ^things we have done without Psych 
Services . It 1 839 ] hit us , and we didn^t have 
the people. It was necessary that we have super- 
vision . . . interpretation of the law was a big. 
thing with us on that. 



Loudon County : 

Loudon County hadn't done too much for com- 
pliance. We weren't geared up to do it. Then, 
through the Co-op, we could better meet re- 



quirements. 



# 



Lenoir City ; 

We not omy meet the state requirements, but I 
feel comfortable with it. The fact that Jerry is 
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^in the background , . , in hearings • . . his expertise in 
general . • . Everybody has a high point with the Co-op. 
That's mine . . . Theae people will be at my side if I go to 
court. ' 

With the implementation of the Psychological Services 
Program also came a significant shift in financial structure 
of the Cooperative. ^The member districts who had pre- 
viously subscribed on a service-by-service basis were now 
obligated to aa^across-the-bbard committment to share Co-op 
administrative and clerical costs. 



As Psychological Services moved to a predomiAent role 
in the program and funding of the Co-op, a shifif in leader- 
ship occurred, as well. Jerry Morton increasing^ly became 
the central leadership figure of the Co-op, with Bill Oakes 
jmoving to a figurehead role until his death in 1976, when 
Jerry Morton was officially appointed as LTVEC Director. 
The dominance of Psychological Services over other aspects 
of the Co-op was reflected iti the Board's (^ledsion to main- 
tain Jerry Morton as full-time Director of Psychological 
Services, and t(!% dimply tack on an additional $3,000 to that 
salary to cover additional responsibilities as Co-op director. 



The districts began to experience benefits from member-' 
ship in LTVEC, as the Co-op increasingly enjoyed a ..surge 
of rene*/ed credibility^nd fiscal feasibility. In an interest- 
ing new twist, an outside intrusion (Public Law 839)', in 
which promises by the outside agent were kept, a recipro<iaI 
, relationship was shaped which benefitted the districts and^ 
thereby revitalized the Co-op. Or as Mac McDowell of 
Monroe County puts it, "Public Law 839 became the lite- 
saver for financing the Co-op, but the Co-op was the life 
saver for the school systems." 



LTVEC: CIRCA ^79 



En'^ering the LTVEC milieu for an eighteen-day on-site 
visit in early March of '79, the obsei'ver wis suddenly, im- 
mersed into a flowing collage of people, events, places, 
attitudes, ^ dialogs, joys, disappointments, and even a few 
dull, empty moments — a slice in the life of an educational 
cooperative. 
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' Through these days, the Little Tennessee Valley 
•Cooperative was busily about its routine acitfvities in the 
slightly updated locker room area of ' the oQ^ Everett High 
School gymnasium on Evferett High Drive., Psychologists 
moved in and out of the building, took phone calls, artd 
huddled in twos and threes to ^ discuss special caseg. 
Phyllis, Sector of the EnvirQnimental Education ProjeS",' 
hurried about planning an upcoming three-day retreat for 
project teachers and adminisltrators . Representatives from 
the Co-op miimber'" school districts^ and other neighbor 
districts— Knox County, Oakridge— 'convened to plan for the 
summer Gifted Program. The Board met, talked, decided 
that special snow day curriculum packages would be a super 
idea, determined that .ALOCA must maintain its >admin- 
istrative cost responsibilities to the Co-op, and heard 
Jerry's latest ideas for restructuring the budget and for 
developing a Handicapped Education Center. The staff met, 
reported problems and progress, and exchanged war stories 
from their schools and projects. Gynden created budgets 
and alternative budgets for 'Tg-'SO. And Jer^y provided 
direct leadership and support for all of these internal 
activities, while also communicating with TVA and other 
external agencies, and engaging in the yearly springtime 
negotiations with districts in deciding upon levels of pro- 
gram and budget commitments. 

Th\r-j^urpose of the site visit had been to document a 
regional environment learning system, a network engaged in 

i^environraental education of some description— either formal 
or informal, in community or in schools. Early into the 
case study, it became evidlJnt that LTVEC would fit no 
orderly dr "tidy definition of an environments educational 
learning system. The Co-op was a network, and had to do 

-with environmental education, but it 'had little in common 
with designed models and was, instead, a prime example of 
an organic, naturally-occurring system. 



Observations of current operations of the Co-op in 
light of its history,, as reported in earlier pages, have 
thejfefore led the ob^rver to make a case for LTVEC as an 
alternative en vironmentc^ education network, one which has 
utility in conceptualizing the nature and e?volution of net- 
works, and which seems^.„gjjgo to enrich current definitions 
of environmental e^ucation7^^"^^i^]^ 



LTVEC is a network involved direcj||]y and Jndirectly in 
environmental education of three types: 



(1) The changing consciousness of" the. psy- 
chosocial environment, 

(24 The restructuring of the political en- 
vironment. ^ 

(3) The increase in individual and collective 
understanding of a- responsibility for issues 
related to the physical' environment ^ 
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y The Psycho-Social Environment . Under Jerry Morton's 
leadfership, promoting change ST the psycho-social en- 
vironment — particularly in schools, -but also in the Jarger 
region — has been the primary target of the Cooperative's 
effort. Jerry came to the Co-op with an academic history 
in school psychology and a deifep personal commitment to 
humanism and social ethics. For him, the greatest needs 
facing school districts and the people of the entire Little 
Tennessee Valley were needs for more nurturing and sup- 
portive environments, wh^e individual needs could be met, 
and especially where the grbwing subculture of handicapped" 
children englemig, to the three-county area could begip. to be 
remediated. 



The outreach psychological services program had been 
the first major thi*ust in this direction. Psychologists first 
ieaht^ students for handicapping conditions, assisted in the 
identifk:ation of the child's needs, and participated in the 
planning of an individualized educational program (lEP) for 
each cnild, a program that would be essentially within the 
mainstream of the system's overall educational program. At 
the time of- the case study observation, psychologists had 
becom^' increasingly involved in the implementation of the 
lEPs and were being further utilized as consultants for 
special problems of teachers and administrators. Problems 
related to the promotion of affective growth ahd develop- 
ment of individual .-children, classroom climate, com- 
munications with parents, ahd even, discipline problems now 
often come under the influence of the psychologists. 



Schoftl districts are under constant threat of suit since 
^the passage of state, and> later, federal laws regarding 
handicapped education . Note , for example , the following 
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"case discussed in a recent educators' newsletter (University 
ot Tennessee, College of Education, March, 1979): 



A recent article in the ' Cleveland 
Daily Banner reports a 



(Tennessee) 



^ _ . - due process hearing 

decision involving a 14-year-old autistic child. 
The hearing officer l-eported a four-^art decision 
as follows : 



—'That the' Cleveland City School Sydtem is 
-attempting to place Jeffrey Wade Rayburn in a 
specikl education program which is inappropriate 
to his condition and need.' 

—'That the Cleveland City School System is 
"denying educational services because no suitable 
program of education or related sdrvice is main- 
tained.'" 
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That VValden House is an appropriate placement 
and that Jeffrey Rayburn should be placed at 
Walden House as soon ai possible' 

--'That Walden House should provide . reports and 
all other data or procedure considerations to allow 
the Cleveland City School System to conduct 
monitoring activities as mandated by law and rules' 
and regulations of the State Department of Edu- 
cation. Program must be reviewed annually.' 

Walden House is a private agency in Nashville 
which specializes in care for autistic children. 
Evidently it is the only such facility wi'thin ^he 
state. The' cost to the Cleveland City^ School ^ 
System will 5^ $18,000 per year. 

This is an example of the tremendous impact of 
PL 94-142 and Section 504. Some would argue 
that local schools cannot stand this type of ex- 
penditure. Such an argument has some validity. 
The argument could also be set forth that if this 
child could have been identified at age three and 
appropriate placement in the least restrictive 
environment started eleven years ago, 4hat costly 
residential care would not now be necessary. The 
latter may be speculation. Nevertheless the 
pre^nt law is real and individuals working within 
the^chools must be able to recognize signals and 
\o identify handicapped conriitions. Autism is a 
low incidence handicap; and many of us would 
know nothing about the cause, symptoms or 
required treatment. * 
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It seems essential that we work deligently to In- 
corporate appropriate matericd in our courses and 
prograr^ so that the graduates of this College 
may be able to cope with even the most unusual 
of handicapping conditions. 

— WilMara H. Coffield, Dean. 

According to Public Law 839, districts were liable if 
the special needs of students could not be met. Here 
again , the cooperative provided solutions or partial solu- 
tions to the problemt First, Jerry Morton gave the dis- 
tricts a language for understanding and articulating legal 
and programmatic responses to the needs of handicapped 
children. As one person interviewed described this in- 
fluence, "Jerry has brought about remarkable change in the 
values and attitudes of superintendents, principals, and 
others in the schools." 



Still the districts had few avenues for becoming more 
responsive to special needs. The Co-op assisted by begin- 
ning to develop speciality programs. At the time of the 
site visit, in addition to regular psychological service, the 
Co-op had become instrumental in several additional pro- 
grams which were also impacting the psycho-social en- 
virofiment of the region: 

Handicapped Child Development Program 

Class for hearing impaired 

Physical therapy and speech therapy services 
Program for enotionally/hehaviorally disturbed 

giftefl students 
A preschool child find program for locating young 

handicapped children 
Program for gifted youth. 

Through the efforts of LTVEC, the consciousness of 
the districts is obset^i^ably sensitive and atypically advanced 
in the area of affective education, and according to reports 
of superintendents, 'principals, and teachers, the 
psycho-social environment in schools is much irriproved over 
earlier years. In short, a language and action of caring is 
being spoken through the LTVEC network, as reflected in 
one teacher's response to the question, "What are you doing 
in your school?" — "Trying to care, I guess." 

And this plea for meaningful changes beyond schools, 
and into the larger psycho-social community environment is 
being voiced more broadly, as Jerry Morton appears at 
hearings on ' the future of the Tellico Dam Project, and 
writes to David Freeman, the new chairman of the TV A 
Board: 



While we are p-roud of LTVEC's record and the 
cooperative" spirit of our communities, there is a 
limit to what the school systems and communities 
can accomplish when addressing regionad issues 
with local resources. LTVEC is identifying and 
reaching some of those limits. The^list ranges 
from generalized problems to detailed specifics. 
For yfexample, (1) we nted a centrally located 
Spedal Education Center (Vonore area), (2) we 
nee;9 a coordinated effort among all human service 
agencies to effectively serve the high incidence of 
adolescent pregnancies to include assistance to 
the surprisingly large number of young women 
(12 to 17 years of age) who decide to keep their 
babies, (3) we need to establish alternative 
school-type programs for the emo- 
tionally/behaviorally difficult student who is 
constantly being expelled or absent from school, 
(4) we need more, effective adolescent gyne- 
cological services to cope with the diagnostic and 
follow-up work for the hi^h incidence of sexual 
child abuse being found, (5) we need region-wide 
planning and strategies to cope with the 
over-crowding of schools as k result of people 
moving into the area, (6) we need more com- 
prehensive mental health services to cope with the 
ever increasing emotional crises our population 
seems to be experiencing as a result of fhe pres- 
sures created by new people moving in and con- 
tinual demands to change life styles at an ever 
increasing rate, (7) we need region wide trans- 
portation planning to cope with the tourists, the 
cost of gasoline, and busing of children to 
school, (8) we need a regional approach to cope 
with the lack of or over abundance of medical 
services to specific communities, and (9) we need 
a regional approach to water usage, waste dis- 
posal, and industrial development.' 



The Restructuring of the Political Environment . Net- 
work s~are~TypIc£dTy~pol^^ 

formed when smaller, individual units judge coalescence to 
be somehow desirable, profitable, or beneficial to their 
individual interests (i.e., interests In money, power, ideo- 
logy, etc.). Networks, once operating, are maintained 
because the^ continue to serve the .political interests of 
participating parties, because they are reslstent to change, 
or because they are oxternally controlled. 
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The impetus to the formation of LTVEC was initially 
very powerful. . As described earlier, two forces en- 
couraged this coalescence: (1) the threat of impact of the 
model city upon the region and its* school districts, and (2) 
the promise of money to accompany this impact. At this 
phase of the Co-op, however, the actual cooperative ex- 
change was little more than shared planning . As ex- 
pectations for Timberlake dwindled, another political con- 
cern replaced earlier ones — the districts became in urgent 
need of means tQ implement the Handicapped Education Law. 
Now re^al collateral entered the network exchange — to in- 
dividual districts went psychological services ^ legal aid, and 
Consulting; and from the districts into the maintenance of 
the cooperative went financial support. In addition, other 
services came available for very little additional cost . So 
for a long whilo., it was much a story of the fishes and the 
loaves V The districts gave a ^ttle , and received much. 

To say that there was a cooperative educational net- 
work in existence over the past nine years is not to say 
that the primary mpde of political activity of the various 
districts wa^ a cooperative one. In fact, their investment 
in cooperative exchange was relatively slight, impacting less 
than twenty percent of their students, and for the most 
part, they continued as solitary, individualistic political 
federations cpncerned with local problems, and meeting them 
with parochial views and solutions. 

Then and now , the importance to the * districts of 
autonomy before cooperation was realized by Jerry Morton, 
and clearly established * as part of the LTVEC rationate . 
Note the following quote from a recent LTVEC proposaJ: 

Several :^ears ago seven school systems in 
Southern Appalachia decided to work together on 
special projects to enhance their services to 
children without sacrificing their, individual auton- 
omy. 

It is important to note that each of the school 
systems had and has its> own unique way of 
making decisions, implementing policies, and 
concepts about its educational goals. Their 
commonality lies in their desire to serve the 
children in the best possible way and the com- 
mitment to cooperate, when possible , with each 
other. 

Still, formal and informal network interaction was 
steady over the years, and the Cooperative became a stim-^ 
ulus for ongoing political discussion and collective political 



concept development related to a wide range of educational 
and public issues. The legal rights of children and par- 
ents; building concern and establishing local action on the 
question of tax base, school consolidation, career education, 
programs for the gifted, TVA*s responsibility to the re- 
gion — all entered the arena of network dialog in one form or 
another » And so, in addition to an exchange of services 
for dollars, LTVEC was also operating to provide an op- 
portunity for the development of a shared, sophisticated, 
and unified political consciousness* 



On the surface the districts* continued participation in 
the Co-op is strictly of a utilitarian nature. According to 
one superintendent: 

"As long as the Co-op is providing services and 
is cost-effective, it will be supported. When it is 
no longer c(j||f' effective , it won't be supported." 

Still, there are indications of a deeper network oper- 
ating among the seven districts, a network based on un- 
spoken person-to-person commitments^ a comraderie , built 
upon empathetic understandings, on shared problems, on a 
sense of fraternal belongingness which strengthens group 
cohesiveness and resists external infiltration , and on a 
shared admiration and trust in its leadership figure, 
Jerry Morton. Board members make occasional reference: 

The Co-op is a common meeting place. I begin to 
see the problems of Floyd P. over in Blount 
County and so on . . . Also, Bill H. I would 
have never known him as well as I know him now 
' if it hadn't been for the Co-op. Now I feel freer 
to talk with him. 

***** 

The Co-op has encouraged my awareness of other 
people's question^ land problems. 

* *| * * * 

It's helped me just to get with the group and 
share common problems. That's a big thing. 

***** 

If Jerry remains director of the Co-op, our 
district will stay in. 
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Thus LTVEC has contributed ijtli^ectly and indirectly to 
change in the political environment of educational systems in 
the region. It has been instrtimental in the development of 
political network r has been a catalyst to the development of 
shared political constructs, and has concurrently con- 
tributed to an underlying personal network which brings 
strength, resonance and resiliency to that political network. 



Current EnArironment Issues . Probably no otheiMfegion 
hi this nation has been more consistently engaged in' wfiat 
we are currently naming "environmental issues" than have 
the people of the 'Little Tennessee Valley. Their birthright 
to roots along the Little Tennessee River brought with it a 
birthright to constant impingement upon their land and way 
of life by external forces which claim some alien rights upon 
the iliver and its resources of energy and^eauty. 

s 

Even now, i visitor to the area is left with the feeling 
that in the early stages of Tellico talk, the upheav^ sur- 
rounding the project was radical and intense across the ' 
region, in communities, and within institutions, including 
school districts. People were pulled between dreams of a 
Camelot an!!!^ dread of another Oak Ridge. In. many cases 
deep cuts were made through the marrow of systems in 
which the support and direction for communities and schools 
were located. A high school principal in Monroe County 
remembers some of the difficulties: 

Well, our Lions Club was a* strong organization 
until one of the members introduced a resolution 
that they were opposed to the dam. But not 
everyone was opposed. So that sort of destroyed 
our Lions Club for awhile . T he same thing 
happened in PTA. We didn't start* it back until 
four or five years ago. It got to a thing even 
your friends didji't necessarily agree with you. 

The struggle, for a time, was one of survival and of 
maintenance of a semblance of equilibrium. It was a time djf 
reactions, rather than initiations; of buying the TV A line 
rather than taking community control . and self-respon- 
sibility. Even LTVEC in its inception was essentially re-- > 
active, not proactive, in regards to planning and de- 
cision-making regarding critical environmental issues. 
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Eventually the smoke screens lifted, leaving the pecjple 
ot the Little Tennessee Valey scarred and cynJfcal. Now, in 
the spring of ^79, the environmental issues surrounding the 
Tellico Project have been resurrected by the congressional 
decision not to dose the dam. Reactions of the Co-op 
member school districts, however, are now more cautious 
and reflect the wisdom of experience: 

Right now it's an emotional disturbance. From 
the beginning it was a split issue, still is a split 
issue. There was a loss of confidence . . . Most 
of us have adopted a wait-and-see policy — we got 
along without it and will get along with it. It took 
some of the more valuable farm lands . Has af- 
fected the roads. We have to drive buses farther 
to pick up children . . . It's kind of a 
no-man^s-land thing now. 

, ♦ + + 

Now you don't hear too much about it, per se, in 
the community. It's sort of an attitude that 
people aren't going to be hurt. 

^ ***** 

It hasn^t had the effect that everybody thought it 
would. When they started buying up farm land 
and moving everybody off, they just moved right 
out het-e to the other side of the community, so 
enrollment hasn't changed muchn 

***** 

It's something we read about in newspapers and 
see on TV. Most of the people in this area are 
for it at this point. Here it, sits. Ninety-eight 
percent complete . . . People think it's stupid for 
$100 million to be sitting over there useless. The 
people against it in the beginning feel it makes a 
mockery of the court systerp. 

■ )f( 3fc % % % 

/ 

I 

The Tellico Prpject didn't affect us. Economi- 
cally^ we'll lose the tax base. If it goes, we'll 
have people, who move into the community. 

I 

Since the Tellico fiasco, among local people i there has 
been a tendency to deny or discount environmental issues, 
and an active distaste for "environmentalists . " Still , the 
people subscribe to environmentally sound values. ' Jerry 
Morton describes this attitude; 



Environment is such a dirty word ' here. An 
/ ^environmentalist is a missibnary who is coming ijni 
\^ and has no idea what is going on, and say«,*^ 
"/You're not gotfig to be able to earn a livixig 
because we're going to save this unique earth- 
worm here . " Environmentalists are outsiders . 
They fly^in representatives from the Sierra Club 
who talk about flying in fishermen from Nevada, 
yet you know they won't arpve, Still, fun- 
damentally these people are extremely en- 
^ vironmentally oriented i 

\ 

One modest effort toward directing this environmental 
orientation is the LTVEC Environmental Information ^nd 
Personnel Integration Project currently funded through a 
grant from the Offic^ of Environmental Education. In a 
recent project report overview and goals we're stated as 
follows: 

^ The overall objective of this project is to in- 
te'grate environmental information into the pro- 
fessional and personal lives and perceptual sets of 
selected high school teachers . and administrators ' 
so that their individual understandings will lead 
them to apipropriate curriculum ranges . This 
objective will be accomplished through a two-part 
effort . First , recognized experts will present 
environmental information to the educators. 
Then, through small-group strategies, the edu- 
cators will be assisted in integrating this in- 
formation into their own persqnal Realities. Our 

•hypothesis is that once the above is accomplished, 
the educators will initiate curriculum adjustments 
to teach their students the newly-acquired in- 
sights as part of their normal subject matter. 
Through these strategies, we hope to overcome 
the traditional difficulty of transforming en- 
vironmental information into professional and 
personal behaviors and ultimately to the class- 
room . 
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In the project , teachers ancl administrators come to- 
gether for Saturday workshops in- which certain "experts" > 
on current environmental issues share information through 
lecture, media, and discussions. In after-school follow-up 
sessions, a consultant with expertise in group process 
facilitation conducts small group follow-up , integration and 
extension sessions. 



The sharing of information is seen as important, and 
certainly curriculum development is desired , as well; ^ut 
the real hope of the project seems to be that it wiU create 
the beginnings of an environmental action advocacy network 
within the regton. The project is very much an extension 
of philosophy and planning of Jerry Morton to increasingly 
engage LTVEC in meaningful involvement In the en-- 
vironmental issues and related regional planning . Bill 
Poppen, the project group process consultant, summarizes 
that motivation: 

He (Jerry) was concerned about the process of 
, change and how it works^ in this area; how com-- 
munities react to change and how change takes 
place. He saw the project as' training awareness 
raisers rather than curriculum developers. 



Jerry Morton himself has said on one day to the observer: 

We hope the people in the project wiU move from 
\ pacifists into people who are involved. And this 

\. wiU serve as |5)ractice. for them to expend their 
project experience into this new setting. 

And again on anol^^er day, \o project? participants: 

There is a mass of (environmental) information". 
Relating this information to school districts is the 
^ responsibility of these persons in the OEE grant. 

You will initiate and create a process of ^hat 
'kinds of (environmental) planning needs to take 
'"'icje, and in your classroomsT what kinds of ex- 
3es need to lake place for kids to do this 
^of planning. 

, \ 

Mr. Sparics, principal of Vonore High School describes the 
content of those sessions consistently with other par-- 
tiqipants : 




^e've hit on some things/^that were real in- 
formative. It's • things you wouldn^t think of 
finding out for ^yourself . 



ERIC 
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In the sessions observed, dialog was open and active.-' 
Topics of all sorts were introduced^ discussed, haggled 
over, laughed about, then set aside for pth'er topics. 
Snatches from the discussions indicate somq^thing of the 
nature of the whole: 



(About mass transit and new expressways.) 
That's just feeding the problem. You're talking 
about laying down more pavement for these cars. 
It's ^just like the world food problem — the more we 
feed these people, the more there will be. ' 

Maryville and ALCOA don't really woVk t6^ether. 
Maryville needs the water system Alcoa h^sl Alcoa 
needs the sewage treatment system Maryville has. 

Vm gonna fill my truck 'up just as long as I can, 
and when it runs out, me and Jimmy Carter are 
gonna walk home. 

If unused land of Tellico were reforested, it 
would probably proVide more energy than the 
dam . / 



Teach them to read, write, and cipher, and work 
the hell out of them. 



I need to know how to work with children who 
come from homes that have no love. 



Ralph Nader has said our schools have more 
problems than thfey should * toler^tte and more 
solutions than they use. 



/ = Final Thoughts 

' The case study of the Little Tennessee Valley Edu-- 
cational Cooperative falls far short 6f providing an arche-- 
typal Regional Environmental Learning System . It does, 
however, seem to offer several insights into the nature of 
networks and networking* Observations from LTVEC and 
the Land Owneri* Organization remind us of that whiclf we 
already knew— networks are little more yet nothing less 
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than human processes, complete with all the foilSles and 
equally repete with unlimited and immeasureable potential. 
Networks are observably more organic than mechanical. 
They are complex, non-linear, resilient, yet fragile. They 
have lifetimes, and must germinate and mature before be- 
coming fully functional and productive. They demand time 
and space for growth and their development charts man- 
dering pathways to strange cadences. 

Plastic networks, like the Charrette, cloned overnight; 
pressed, pushed, and artifically rewarded to produce net- 
work-like processes generate plastic, mechanical outcomes, 
and fail as networks. In sum, networks are part of the 
living environmeat, and our desire to understand, utilize, 
manipulate and- control them brings the same gains and 
risks from which we have sometimes learned and too often 
failed to learn in painful lessons through previous ex- 
periences with control of other segments of our living 
environment . 



/ 
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THE CITIZENS LEAGUE 




In every dty from time to time citizens join together to 
study environmental and other social problems . Most 
groups last a few months , some a few years. In Minnesota 
there is a metropolitan study group that wiU soon celebrjttQ 
its thirtieth birthday. It is the Citizens League of 
Minneapolis-St . Paul . • * 



When the Citizens Program Committee met in January, 
the following were among those being considered as pos- 
sibilities foj the 1979-1980 research yeaV. 



08) How to select sites for undesirable en- 
vironmental» uses, such as power plants^ 
power lines and solid waste disposable 
facilities • 

(109) State policy toward copper-nickel 

mining, including whether to encourage, 
method of taxation, and type of en- 
vironmental regulation. 
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110) 'Regional water policy, including 

assuranoe of good supply of surface 
and underground water in Twin Cities 
area in coming years. 

111) Pricing oV eiectrical use and other 
energy soii^rces. 

112) Potential of alternative systems, such as 
solar ^ energy, wind power; and internal 
sewage systems. 

\ 

113) Planning, ow^iing and operating district 
.heating facilities. 

114) Desirability of mtind^tory vehicle 
emission and safety inspections. 

115) Area-wide policy to encourage and 
facilitate recycling of solid waste. 

116) » Stimulating the building and main- 

tenance of cities which are both human 
and beautiful. 

# 

117) How to reduce energy usage withoLjt 
negative impact on' jobs and the econ- 
omy. 

W^l Planning for energy usage in Minnesota 
over the next decades. 

119) Optimal use of land around lakes in the 
Twin Cities area. ^ • 

r- ' 

120) Pollution control standards on com- 
mercial and industrial development. 



Unfortunately ^or environmeritalists , there were l94 
other topics bidding for attention. By May the list had 
been reworked and reduced to twelve. Two environmental 
topics, were still among them. These were: 

(1) Selecting Sites for Undesirable Environmental U^es 

(2) Designing a Pricing System to Promote Energy 
Conservation without Placing a Burden on Low 
Income' People. 

Aad of course, it is reasonable to expect that several other 
Study Committees will get into en vironmental. issues as they 
do their work. 

V 



In identifying possible stu(5y topics, the League com- 
mittee had heeded admonitions in the Fitzgerald Report of 
1974: 



There must be a fundamental re--thinking of the prO^ 
cess by which our work projects are selected. Input 
must be broadened. Ways must be found to add 
projects throughout the year. Others in the com- 
munity should be consulted with more fully , • • and 
the results of our appraisal of community problems 
should be more fully shared with these other organiza- 
tions . (p. 4) . 



In the League office in the #j^yndicate Building in 
downtown Minneapolis they keep a me of ideas from news- 
letters, journals, and the media. Exchanging letters with 
public officials and talking with research^ and planning 
people are also ways of gathering ideas fpr possible topics. 
Weekly breakfast forums generate other possibilities. 

The Program Committee uses certain criteria in de 
ciding on the final list of topics for study. Some of these 
criteria arer importance and urgency to the community, 
cost benefit, the probability of recruiting volunteers, the 
prospect for implementation of the recommendations which 
might be made, awareness of the public in the subject and 
the prospect of ^ the issue being settled by reasoned analysis 
rather than by emotional factors. 

/ 

THE WORKINGS OF A STUDY COMMITTEE 

■ ^ 

Each Study ComrmH^e is comprised of citizens who 
have paid the twenty dollar mem bet-ship fee to join the 
League. Many join just for the privilege of participating in 
one topic's deliberation; some join just to receive minutes of 
weekly meetings. On J:he sign-up form the members tell 
some characteristics about themselves, including any 
"special interest" involvement. If it appears to the Program 
Committee that there is an "imbalance" in the newly-forming 
committee, it recruits additional persons likely to diminish 
the imbalance. ^ 

The Study Committee convenes (or a long ev>ening^is 
meeting every week, usually for the better part of a year* 
One of the members serves as chair. The greatest share of 
the work is done by a staff member ^employed by the 
League. Most of these staff members are former news- 
papermen, well acquainted with the history, organizations 
and activities x^f the community, and skilled at writing a set 
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ot minutes that is terse, cogent, complete and about as 
interesting as minutes can get. For example, from the 
Committee on School Desegregation's April 17 minutes; 

.The meeting was called to order at 6:40 p.m. by 
chairman Stephenson. Minutes of the meetings of April 
10 apd March 27 were distributed. ' 

Ex perience of the Gr^hek Family -The chairman intro- 
duced May Lou and Carl Grahek and their daughter 
Julie. The Graheks havf lived ^ in the 
Lexington-Hamlirie Community for twenty years. They 
are a- White family and their children have attended 
integrated schools. In' fact, they were attending 
predominately minority schools before the desegregation 
occurred ... . 

I 

The staff member makes the many arrangements needed 
and. in successive revisions, prepares drafts of reports the 
Study Committee will submit to the League's Board. 

One of the things that a Study Committee does first is 
to acquaint itself with the facts of the is3u6 under con- 
sideration. This is done chiefly by inviting resource per- 
sons (the scholars, -the aggrieved, the greatly experienced) 
to appear before the committee to present information and to 
discuss matters with members. 



Some years ago. one Committee was studying "en- 
vironmental choice." more particularly, the mechanisms for 
resolving issues and conflicts in the use of Minesota's 
environment. The following would be typical (though it did 
not actually happen) of the discussion one evening: 



Resoirt-ce person: I want to express my opposition to 
creating the Office of Environment Assistant to the 
Governor. If the Governor someday is unsympathetic 
to environmental concerns, he would appoint someone 
who wiU give us lots of trouble. 

Staff member : Do you .favor an official who is 
semi-independent, possibly totally independent of the 
Governor? 

Resource person: I don't know. It depends on how 
he or she would be appointed. 

Committee member : The C^overnor's Office is the most 
visible spot in tlie state. That's where the power is. 
And that is where people see what is going on. Of 



course it can be misused • But isn't that where people 
can best see what is agoing on? If we happen to get 
an "anti^-environmentalist'' for our environmental assis- 
tant, isn't it best to have him out in the open where 
we can see him? 



Committee orientation may take the form of seeing 
things for oneself. A few years ago a Committee studying 
the "use of the Mississippi riverfront" undertook two boat 
trips to gadn a first--hand view* It travelled on the 
Jonathan Padplford to see the riverfront from downtown St • 
Paul to a point just south of the Lake Street bridge. Later 
the Committee viewed the river between Grey Cloud Island 
and downtown St. Paul on a towboat furnished by the J. L. 
Shiely Company. 



THE ISSUES 



A summary of testimony, observations and discussions 
becomes the first draft of Committee findings. The Com- 
mittee usually spends a month or more preparing findings, 
going- through two or three drafts. 

A Committee that .studied the "control of Dutch elm 
disease" presented findings such as the following: 



1^ 



Public interest in Dutch elm disease has increased in 
direct proportion to the incidence of the disease. « In 
1976 public interest in Dutch^ elm disease became 
higher than ever before . Large red paint marks were 
used to identify diseased trees, and because so many 
trees on st;reets and parkways were discovered; the 
public naturally became aware. . . 

However^ it is not clear if high public interest will be 
sustained. If in fact the Twin Cities area is successful 
in curbing the rapid spread of the disease, then trees 
will remain alive longer. ft is possible the result 
would be a drop of public support for^a control pro- 
gram. » Syracuse, N.Y., for example, had a first rate 
control program from 1957-1964 , but then public sup- 
port waned, the disease took over, and almost all elms 
died. 4 



In the light of such findings the members of that 
vStudy Committee became increasingly concerned about the 
issue of public awareness and understanding. After a 
number of discussions the chairman of the Committee surH- 
med up the consensus of opinion among members in the 
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following statement (included as one of the conclusions in 
the report) : , . ' 

Unfortunately, although public awareness is high, 
public knowledge of what to do about the pr(4yem is 
woefully inadequate. Misinformation may be worse than 
^ no information at all, We must find A way to correct 
information conveyed to the public and to public 
leaders. 



The Committee went on the discuss specific steps 
to be taken to remedy this problem. After careful 
deliberations the members of the Committee reached a 
Unanimous' decision: 



We recommend the legislature set aside a significant 
portion of state funds to finance improved public 
education, with the provision that a portion of the 
education funds be earmarked as matching funds to 
qualified private groups. | 



\ The final testing ground of a report, of course, is the 
community itself. Hence League officers consider it im- 
portant that understandings of the problem, and its solu-. 
tion, must be concurred in by a broad range of organ- 
izations and individuals, both public and private. A Com- 
munity .?Information Committee takes on this role of com- 
munication with the community. 



Environmental education situations change quickly, of 
course , and even a new report can be out-of-date . In 
such instances, the Community Information Committee may 
suggest to the Board that another study be programmed on 
the same issues. 



Since 1952, the year it was founded, the League has 
iesued recommendations on nearly 300 matters of public 
concern. "Whenever we do a report," says a League board 
member, "somebody at the legislature is ready to sponsor 
the results." In fact, many League recommendations have 
been adopted into law. Equalization of School District fi- 
nancing, establishment of an urban regional governmental 
body , the Metropolitan area, and the sharing of medical 
facilities between a public hospital, Hennepin County Gen- 
eral, and a private one, the Metropolitan Medical Center, 
are among the many League recommendations adopted into 
law . 



/ 
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"The Citizens League has come to be recognized as the 
single most effective organization in the whole country," 
says Verne Johnson, a former League Executive Director 
and President. In the words of John Yngve, a former state 
legislator, "I doubt if there^s another organization that 
comes close to the effectiveness of the Citizens League in 
its caused arui orientation." 



The League is not without detractors. Some critics 
see the League as an organization of liberals and elitists. 
One local businessnan was quoted complaining that "it's a 
downtown lawyers' group." 

There are, in fact, a good number of attorneys (from 
'the suburbs as well as downtown) — but a similar number of 
university faculty members and former elfected officials 
among League members • According to a 1977 article in 
Corporate Report , some critics of Ihe League have pointed 
out that of the 3,000 individual members, only 450 are from 
St. Paul and of the corporate membership from St. Paul, as 
few as 100 out of 600 are from St. Paul. Women, minority 
groups and low income citizens are, according to these 
critics, also poorly represented in the League. JThe critics 
draw attention to the fact that about a third of the 
League's 36<^member Board of Directors are businessmen and 
a quarter are attorneys. The remaining membership is 
composed of government officials, university professors and 
others. League documents indicate a continued awareness 
that the Committee purview may sometimes be limited. (See 
more on this in the next chapter.) 

The League has changed, of course, across the years. 
It has slowly evolved into an area metropolitan organization 
concerned with all social problems. In the beginning it 
concentrated on government initiatives to raise taxes, 
particularly for financing public buildings. It reviewed 
candidates for public office. In subsequent years, it 
stopped evaluating, candidates and assumed its present focus 
on issues. 



The Citizens League is a non-^profit organization. Its 
e^iMKs are financed by 90 of the area's 100 largest private 
arj^public companies, by 500 smaller businesses and by the 
twenty dollar membership fee. 

Since 1974 thn League has bnen increasingly involved 
in studies of community systems that are heavily 
non-governmental, such as health care, housing, trans-- 
portation, public safety, and education. 

{)0 



Environmental issues have? been given special attention 
in recent years. As indicated eaHier, thirtyn environmental 
topics were among the 207 possible topics considered for the' 
1979-1980 research year. Two of these topics were included 
in the semi-final list of '"twelve topics. This may be an in- 
dication of growing awareness of citizens living in urban 
areas for the ways environmental problems affect their 
lives. ^ As the Citizens League moves into its fourth decade, 
it demonstrates again and again that citizen study groups 
can deal effectively with complex Social problems. * 
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OPPORTUNITIES FOR EVALUATION 



In this portion of the RELS Evaluation Manual we will 
^^Viggest some typical evaluation opportunities to be found in 
Environmental education projects. In the fifth chapter we 
will consider how a Unit might be organized to undertake 
evaluation studies on a regular basis . The purpose of this 
chapter is to describe some of those needs and to suggest 
what might be done. 

V 

We will continue to emphasize the problems and context 
of program evaluation inquiries by continuing to discuss 
LTVEQ and the Citizens league. From here on however the 
story is fictitious ♦ We will be imagining what, the people in 
those two situations might do to evalu^^ite. There is a bit of 
risk here in misrepresenting those two projects. We (and 
people there agreed) that it is worth the risk in ort^er to 
present a realistic situation for describing program eval- 
uation opportunities ♦ 



Illustrative Application #1 



AN EVALUATION STUDY OF DECISION-MAKING 
Focus; The LTVEC Board of Directors 

/ 

Imagine that the Tennessee State Department of Edu- 
cation has recently been considering funding the Little 
Tennessee Valley Cooperative and might establish two or 
three additional cooperatives for educational services, the 
State Department decides to evaluate LTVEC formally • 
Informal evaluation in past years has led the Commissioner 
of Education to believe that LTVEC could serve as a model 
for new cooperatives. He feels » however, that they need to 
be cautious about how decisions are made in a cooperative 
among autonomous institutions. 

•» 

't 

The Commissioner's staff suggests documenting and 
evaluating the decision-making and policy-setting process »of 
LTVEC . They feel that organizers of new cooperatives 
might benefit from seeing how an existing system functions. 
If there are hazards to be avoided they want to recognize 
them during planning and proposal reviewing sessions. The 
Commissioner is specific about wanting recommendations for 
improving LTVEC to be included in the final report. 

A Blue Ribbon Panel. Representatives of the State 
Department of Education met in January with the Executive 
Director of LTVEC. Together they decided to commission a 
panel to conduct .the LTVEC evaluation. The panel would 
consist of distingmshed citizens, at least some of whom are 
known for contributions in fields related to environmental 
education. Four individuals accepted the Commissioner's 
invitation: 

Daniel Lounsbury, U of T Dept of Sociology (chair) 
Creed Gilpin, retired, formally State Court judge 
Eleanor O'Day, school superintendent and naturalist 
Mary Evelyn Wiggins, Community Services Director, 
ALCOA . C 

^acK wiis offered an honorarium of $1000 and expenses of 
course were to be covered. Total costs were not to exceed 
$7500; A graduate student, Jill Hubka, was employed as 
Staff assistant to work 3/4 time for one month at a cost of 
$600.. . 

6d 
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1^ 

A list of 15 questions were developed to serve as a 
basis of i^iquiry : 

. (1) What is the formal governance structure of 
the Co-"op? 

(2) V/Kat are the patterns of the decision-making 
process? ^ 

(3) How do issues get on the Board of Director's 
agenda?^ 

(4) How are new programs conceived, decided 
on, and developed? 

(5) How are alternatives generated? 

(6) How are policies set? 

(7) How does LTVEC define itself? 'Does this 
affect J^e decision process? 

(8) How are conflicts at the institution level 
resolved? What are some of them? 

(9) How does public opinion affect policy and 
program decisions? 

* 

(10) How are decisions evaluated? 

(11) Is there formal review to decide on program 
continuation and modification? 

I 

(12) How do financial constraints affect the de- 
cision making process? 

(13) What changes would various people like to 
see and why? 

4 

(14) How do individual Boards of Education affect 
the decision making process? 

(15) How, if at all, has the State Department of 
Education, influenced LTVEC? 
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The panel meets for the first time in the Office of the 
Commissionejr with the Co-op Executive Director and several 
of its Board of Directors present to discuss th^^ need for 
information and cooperation. Later that morning the four 
panelists meet to devise a plan of action. (They had 
agreed to complete the job in four weeks, to make their 
report by April 1.) The following chart illustrate^s the 
three main methods of inquiry decided on. 



Method of Inquiry 



Data Source 



Observation 



(1 
(2 



Co-op Executive Board meetings 
Co-op planning commission meetings 





(3) 


Co-op stair meetings 




(1) 


C©~*on Board mpmhprs ' 

^ ^ J 




(2) 


Co— OD Executive Director ^ 


Panel 


(3) 


Co-op project directors f 


Interviews 


(4) 


otaii members 




(5) 


Principals ^ • 




(6) 


Consultants f 




(7) 


Co-op Executive Secre^ry 

/ 




(1) 


Official records of past Co-op meetings 




(2) 


Past evaluations of the Co-op 


Document Analysis 


C3) 


Press releases by the Co-op 


(principally by 
aide) 


(4) 


Booklets developed describing the Co-op 




(5) 


Newspaper clippings 




(6) 


Budget reports. 




(7) 


Funding proposals 



Each panel member was to be involved with each 
method of inquiry. They establighed an expectation of 
continuous communication among themselves so that they 
could verify information collected and discuss problems, 
concerns, and generally shape the direction of the evalua- 
tion study as a group. 



Panel members agreed they needed lots of information 
to consider the 15 questions. Standardized instruments 
were not seen to be useful because there were too few 
persons to contact (and too little time and money to devise 
good standard forms). The ^anel members would have to be 
the data gathering instruments. 

They acknowledged that Professor Lounsbury was more 
experienced in data gathering of the sort needed. He 
agreed to coach them and said: 

A panel such as this is not usually obligated to 
provide others with a good record of what ie going on. 
In this instance are asked to document LTVEC 

decision making, but we can hope to do thij^ only in a 
very general way. For us to understand |LTVEC as 
best we can we will want to record the m£st salient 
facts and ideas. Let us each agree to keep i notebook 
with entries that some one- else can make sense of, and 
to share those notebooks '(or phpfocopies) regularly. 

He suggested that they read Richard Brandt's book. 
Studying Behavior in Natural Settings and made photocopies 
of Chapter "Measurement Through Observatifonal Pro- 

cedures" for eaeh of them. At thei^ next meeting Judge 
Gilpin brought Arne Trankell's book, The ReliabUij^y of Evi- 
dence , to help the/5ri thinlc about that.""" i 



0^ 
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Observation , Observations were used to gain first 
hind knowledge of decision-making activities of the Co-op, 
Events were recorded naturalistically in the observers* 
notebooks. Using tape recordings and the notebooks, one 
meeting was summarized by Jill Hubka in the following 
manner: 



The Co-op staff gathered around the two large con- 
ference tables in\ the back office of the Co-op's fa- 
,cilities. In atten/flAnce were dozen or so graduate 
students for the University ^f Tennessee currently 
involved in an internship with i^he Co-op. The meet- 
ing was run by Environmental E^ducation Project Dir-/ 
ector and the Executive Director V Co-op. The 

latter began the meetiiig by informing the group that 
he had some bad news. He proceeded to eicplain that 
one of the seven school districts the Co-op serves had 
decided to pull out of the Rsych Services program . 
Responding to many question;B, particularly those frorx^ 
graduate students whose jobs ,would be eliminated, the 
Director said: 

Well, think of this in terms of Systems analysis 
and change. We can lear^i from this . whole thing 
about how systems change. I knew I could have 
spent more time with each principal, smoking and 
joking, and maybe this wouldn^ have happened. 
But now we can learn lessons for other systems 
that are similar. One of the overall objectives of 
the Co-op is to^ do the feasibility research and 
develop, but not take over, the individual school 
system . I haven't failed as a person nor have 
. the rest of you as a group. ^ 

A student asked if the Director thought the Co-op was 
eroding. He replied: 

The seven systems think of themselves as a 
support system — as opposed to the other systems 
outlfcdq the Co-op . The seven superintendents 
consistently confer with each other, and sit with' 
each other at meetings out of the counties. They 
are like brothers and clearly see themselves 
together and they feel obligatibns to each other. 

By observing meetings such as these, the panel mem- 
bers were able to see, hear and feel some of the workings 
of the Co-op . The above sequence , for example , provided 
valuable insights on how the Director views the role of the 
Co-op and the relationships holding^ the Co-op together. 



1^ 
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Interview s , The interviewing approach and the 
questions tliemselves differed depending on who was 
being interviewed. Though they did not standardize 
the interview , each panelist anticipated in advance a 
few questions ' to ask, phoned them to Jill Hubka who 
merged them , identified overlap and omissions , and 
prepared a master list of a few key questions and an 
indication of who probably would ask each. The list 
for each superintendent >included the following: 

(1) How do you see the Co-op? What is its 
impact? How does it function? What are the 
most important -events in its history? What 
are its strengths and • weaknesses? 
(Lcunsbury) ^ 

(2) How do decisions get made? (O'Day) 

(3) What are your purposes for participating kl 
the Co-op? Are there some grand , 
long-range goals that it serves? Are there 
political advantages for participation by your 
districp (Wiggins) 

s 

(4) What are some of the critical educational 
problems for your district at this time? 
(Gilpin) 

(5) What do you see as the future of 'the Co-op? 
Should there be more Co-^ps^ across the 
state?; ^Gilpin) \ 

(6y How has environmental education been chang- 
ing in the region? How, if at ,,all, has the 
Co-op^ affected it? j[0'Day) 

Other questior\s arose in. each session, partly to clarify 
responses and partly to, capitalize on the uniqueness of 
each superintendency . ^i^- 

Others were asked similar ques-tions. As many 
panelists as could attend did so. The sessions were 
taped . Usually the sessions were quite informal , as 
they were particularly in the case of the inservice 
training specialist. Her responses led them into 
lengthy discussion of teacher interest in service 
training and willingness to . participate in joint 
school/community projects. 

Informal interviews by Jill Hubka were held with 
teacher participants in an environmental workshop . 
Sh^also held informal interviews throughout the month 
witff the Executive Director , tho he was interviewed 
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formally by the panel too. One interview consisted 
largely of Jill sharing her perceptions. Key ideas from 
the notes and tapes were filed in' a topical file that Jill 
set up. (She had a file folder for each of the 15 
questions plus some other topics such as 
school/community cooperation; energy saving; 
state/ federal regulations.) 



Do.cument Analysis . Documeat analysis served as 
a basis for gathering information on the intents and 
actions of th« newspaper clippings. The following 
exerpts were among those filed in Jill Hubka's topical 
files: 



V^en the Co-op 'was in its beginning stages, public relations was a big 
concern. The Board of Directors agreed that "early visibility should -be 
sought." 

Bo^rd Notes, Sept 1971 

/ 

The Board requested ffom the Director information on public relations, which 
was promptly provided- by the next Board meeting. The document submitted 
as the "LTVEC PR Plan" stated that the objectives of the PR Plan were to: 

A. Acquaint general public with purpose, objectives, 
plans, methods, progress. 

B. Generate personal involvement and commitment to LTVEC program 
by professional educators within affected school systems. 

C. Generate personal involvement and commitment to LTVEC program 
by a significant number of community citizens outside the education 
profession. 

Gain support of a majority of the general public for LTVEC pro- 
gram . 



D 



The report stressed that: "The essential characteristic for success of any 
public r^elations program is an open and sincere approach to the public con- 
cerned," 



Documents, aft ilhistrated above, were used by the 
panel to add to their understanding of the formal de- 
cision-making process. (feneration of the PR Plan and the 
subsequent media corflpaign to promote the Co~op added to 
th\' knowledge of the panel members in attempting tci de- 
termine hovv Boa^rd policy is generated and turned into 
action • 
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Exerpts from the Panel t^inclin^s . The Blue Ribbon 
Panel compiled the data collected through interviews, ot)- 
servation and document analysis into a formal report. The 
various sections of the i^eoort addressed the initial 15 
questions. ^ 



U had become apparent that the State Department of 
' ♦ Education had from time to time made an impact on the 
operations of the Board. Though it had not initially been a 
, key concern, substantial follow-up questioning and tele- 

. ^ « phoning had- been devoted to this topic. The following is 

) an exerpt'from the report: 

t 

«• 



^ The State Department, from time to time, has either made direct requests 
or ottered suggestions to the LTVEC Board of Directors. The changing of 
the composition of the Board is illustrative. The original structure of the 
^Board was created by the Charette and did not include ^superintendents as 
Board members. However, at the end ol the first year of operation, the 
State Department of Education evaluated the Co-op and recommended that 
superintendents be added to the Board. The co-op responded by inviting the 
seven superintendents be added to the Board, but not as voting members. 
One year later the State Department requested that the seven superintendents 
be made voting members through amendment of the LTVEC charter,. The 
Co-op, responded b^ doing just that,. ^ 

Thus, .the State Department has had an impact on the LTVEC de- 
y-ci*ion-making process. The Co-op has been resonsive to the desires of the 
State Department and has been concerned about maintaining a positive image 
and good communications with (the Department. 



Another major topic developed in the final report was 
cost-effectiveness . U; had been stated repeatedly as a 
concern of the Co-op over *the years. A 1973 news release 
stated the Co-op's intent to operate cost-effective pro- 
grams: 

Small school districts lack the resources for all 
educational programs needed by their people. The 
costs of education are well-known, and the money 
pinch has most of us squirming in protest. 
Oj^ew aj^t^ ex p a n d e d u c a t io n a 1 p r ojjjram s and ser- 
vices in a cost- c FrectiVe way is to coo |3erate . 
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In January of 1974 the Coop's director wrote ^he State 
Department ot* EducAtion and assured th%n that: 

LTVEC IS now in the best condition it has been 
since its creation , We are providing superior 
cost-effective programs and services and we 
would certainly like to continue our" services. 



Special education services brought with it par- 
ticular concerns to develop cost-effective programs. 
In August 1974, pne of the problems came ub at^a 
Board meeting: \ ( 

Discussion centered on the prohibitive costs 
of some special ^education programs, The Board 
repeated its earlier concern that LTVEC staff 
members shall explore every avenue in seeking 
alternatives to such high cost programs, including 
searching for ways and means to construct a 
rehabilitation center in the tri-county area. 



Developing cost-effective systems is a concern for 
other than the actual program areas. 

The Co-op decided to employ a con- 
sultant for cooperative purchasing. Those 
systems wishing to participate must do so by 
appointing a representative to the Cooperate 
Purchasing Committee, This committee would 
determine what items, specifications and 
quantities be purchased as well as bidding 
and oth^r procedures to be followed • T he 
/ cooperative purchasing program is to be on a 
limited and trial basis for the coming year, 

(Board^Jiaies, Feb 11, 

iy77) 



The Blue Ribbon Panel decided early on that they 
would not conduct a formal study of cost--ef fectiveness • 
The decision stemmed partly from awareness, that the 
Co-op would shortly be engaged in developing an 
accountability report. TRat report responds to a Ten- 
nessee law requiring the establishment of an accoun- 
tability procedure for co-ops. The bill originally came 
from the Comptroll(*r^s office after an audi t • discovered 
the misuse of a Co-*op^s funds. 



If the Panel had deriderl to measure cost-offec- 
tiveness of a human .service>s (jr^janization^^sTich as 
LTVKC numerous factors other than dollar and gents 
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transactions would have to ho tallied, factors such as 
person-to-person commitments, time spent on shared 
problems, and the sense of belonging to advocacy 
groups. The fact that these values a.re 'impossible to 
measure objectively and accurately does not excuse 
them from considei^'ation . Those people who claim that 
an organization' asking f^ public support is obligated 
to cost out its services are either naiye or ob- 
structionist-^-the most ^ effective organizations are no 
more effective at specifying their effectiveness than 
the least. 



The Pa^el noted that* educational programs are 
developed in a variety of ways. The LTVEC En- 
vironmntal Education program is an example of one 
way. The seeds" were sown in 1971 when a lot of 
discussion at Board meetings centered around plans to 
develop a Human Resources Park. At ^.several times 
during the year various Board members suggested an 
Environmental Center as part of the part: 

The potential of an Environmental Center in the 
Human Resource 'Center was presented. It was 
reported that an employee of TVA with con- 
siderable experience in environmental education 
helped sot up such a center at Bays Mount. It 
was suggested that this center could become a 
^demonstration and model for the whole East 
Tennessee Valley . . . 

(November 1971 Board notes) 

In February 1972 the notion of an Environmental Re- 
source Center (ERG) was reintroduced. This time the 
impetus for an environmental program shifted to fit a 
'national plan: 

A review of the hoped-for ERC was presented to 
the Board . The Senate has recently approved 
and sent to the House a program which would 
establish environmental centers in each state and 
six regional environmental centers in the county. 
In cooperation with the Oak Ridge National Lab- 
oratory, TVA,^ the Boeing Company and the 
University of Tennessee, we hope to' explore 
possibilities of gaining both a state and Regional 
Center for Timberlake. 

# 

By November 1972 it was apparent that Timber- 
hike was fading ffom the picture and that the Co^Cp 
needed to make new plans . In r(^ass(sssing thf^ir [)()si- 
tion, they discussed the position of t\\i) federal govern- 
mont and where monies might be available. 

\ 
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It looks as If the prime stress in Washington for 
the decade of the ^70s will be on career edu- 
cation , environmental education, and the use of 
educational t.v. 



From this discussion followed decisions to write 
proposals in each of the three. Subsequently, further 
discussions centered on the various kinds of programs 
the Co-op could propose. The director was 
particularly concerned about how to fit in environ- 
mental education with the other programs. 

Maryville College is at the forefront of environ- 
mental education in this entire area and they 
operate the Tremond Environmental Center. This 
^ is relatively a new field and none of us really 
knows exactly which way we are heading . Most 
environmental education programs , at the moment 
are aimed at 5th and 6th grades and very little is 
being done in the 9th grade - 1 2th grade age 
groups as well as adults in the Adult Learning 
Center, we believe that we will select this age 
group to be served by environmental education 
and thus cojjnect environmental education and 
adult education. 



It was 1978 before a grant for environmental 
education was actually written. When it was subm\ted 
to HEW on January 16 1978, the focus had shifted from 
Adult Education and a Learning Center to Teacher 
Education. 

This grant will attempt to change attitudes of key 
teachers and principals .through pairing factual 
information and attitude change processes so that 
they will initiate their own curriculum strategies 
in a manner consistent with the needs as they see 
them . It is important to ript need to be aware 
that change is coming and informed decisions need 
to be made to insure that those changes are 
beneficial, or, at least, nonharmful to humans. 



At the ^ugust 9, 1978 Board meeting the Director 
informed Board members that the Environmental In- 
formation and Personcul Integration Project (EIPIP) had 
received the HEW award of funds . By January 1979 
the program was underway. The project director 
reported that (fidministratively and content-wise the 
project is functioning smoothly. A University of 
Tennessee professc^r involved with EIPJP t^epor/^ed that 
as the project progresses, attention vf-iW focifs( more on 
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IKJSsible adjustment in curricula the educators may wish to 
make as a result of what they have learned. 



The development of environmental education as part of 
the Co-op^3 program started with an idea to include a 
center in the proposed Human Resources Park, and finally 
came as a Teacher Education program. Program suggestion^ 
came- from various Board members as responses to Various 
forces, such as the possibility of TVA money, a national 
pr'^^osal to establish environmtal centers , a national stress 
on environmental education, and HEW guidelines for sub- 
mitting a pro posaL From this example, it can be seen that 
the LTVEC programs are shaped by a variety of external 
tactors, and in the end are primarily dependent on funding 
trom new sources. 



Recommendations , The Blue Ribbon Panel completed 
its study ol LTVEC decision by makings sever'al recom- 
mendations • It had previously sounded out reactions to 
tentative recommendations from people of several back- 
grounds and^LTVEC involvements. Arguments against their 
recommendations were considered and some modifications 
made , 



In presenting the recommendations the Panel 
acknowledged that it had not had time nor sufficient wisdom 
to examine all issues and contingencies and did not expect 
the recommendations to be taken at face value by those 
much better acquainted with the history, the resources, 
and the responsibilities of LTVEC. 

Still the panel members felt their reflections were largely'^ 
valid and worthy of study. Thanks, to the cooperation and 
contientiousness ^ of the people of LTVEC and the com- 
munities it served, the members felt they had reached a 
substantial understanding of how decisions were made , and 
left unmade. They concluded thitt by and large the process 
was healthy , neither impersonal nor sel f'-protective , 
admirably reliant on a concerned vigilance, and of course, 
still capable improvement. 




Illustrative Application ^2 

EVALUATING THE IMPACT OF AN INSERVICE TRAINING 
PROJECT 

f 

Focus: The LTVEC Environmental Information and Personal 
Integration Project (EIPIP) 

The intention of this project was to provide selected 
teachers and administration with information about present 
and potential envii-onmental issues in the Little Tennessee 
Vcilley, to help them integrate this information into . their 
own perceptions and values and to encourage them to inco- 
porate new insights on the environment within their teach- 
ing of the regular curriculum. 



The essence of the project was a series of workshops 
in which the participants learned about environmental issues 
from environmental experts and discussed the meaning and 
importance of these issues for their own personal lives and 
the futui'e of the region. It \as hoped that the par- 
ticipants would undergo something perhaps not unlike a 
religious conversion' and even to acquire an evangelical 
commitment to spread the good news to others. Effects on 
professional behavior were seen as a natural consequence of 
this new-found cojhmitment to environmental education 
coupled with a sensejof professional responsiblity . 

These workshops had the following features: 

An eight-hour commitment per month from 
September through May. This involved one 
four-hour Saturday morning session per 
month plus two-hour after-school sessions 
per month. 

Attended by volunteer teachers and ad- 
ministrators sought from all the elementary 
schools, high schools and vocational/ technical 
schools in the group of school/ districts. 

Encouragement given (O^^^^^iit least two 
teachers and one administpator attending 
from each school so as to create an en- 
vironmental education "cell" within each 
school. • 

Saturday morning aossione focused on pre- 
sentations by on vironmQntal experts of 
information about key environmental issues 



(1) 

(2) 
(3) 

(4) 
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followed by small group exercises and dis- 
cussions both to clarify the issues and to 
consolidate the information, T 

(5) After-school sessions involving group dis-- 
cussion focused on heightened awareness and 
understanding of regional environmental 
issues and on increased commitment to per- 
sonal and professjpnal action on behalf of the 
environment* 



The principal expectation of this project was that 
participants would internalize a concern for the environ- 
mental future of the region to such an extent that they 
would initiate: 

.(a) changes in their own personal lives to contribute 
^ toward conservation of energy and resources and 

to reduce human pressures on the regional en-- 
vironment ; 

(b) cAnges in their approach to the teaching of the 
regular curriculum directed at highlighting appro- 
priate environmental issues arising from or related 
to the subject matter (such as through opportun- 
istic references and discussions about the en- 
vironment, e.g., air and water pollution in chem- 
istry; choice of environmental examples and 
problems for exercises, e,g,, graphing population 
projections in mathematics; assignment of projects 
with environmental component, e.g., English 
themes on life in the future, and so on); and 

(c) discussions in their own school district about in- 
cluding more specific: components about the en- 

^ vironment within the school curriculum (such as 
> adopting curriculum units on envirdnmental 
issues, designing integrated studies of the on- 
vironment, arranging environmental experiences 
for students, and so on). 



The conten t of the Saturday sessions was organized 
around four main topic headings : regional economic 
growth; regional population groNvth; regional land use; and 
regional sociological change. The theme behitid ^these topic 
headings was "anticipating and planning the futute.'' Other 
moi\ particular problems ^$uch as energy- use, pollution 
contrV)! and food supplies v/ere dealt with as consequential 
topics ^within the general framework/- 



f KJ 
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An objective of the after-school sessions was to help 
participants explore for themselves the meaning of their 
newly acquired knowledge about the environment ( through 
small group discussions of the potential impact of environ- 
mental changes on their personal lives) in the expectation 
that this would lead to greater personal commitment to 
environmental control and to public debate about its impor- 
tance . 



While the rhetoric of this project espoused the gen- 
eration of a sense of community among educators concerned 
for the environment, this was expressed more in the sense 
of a convocation of believers than of a forum of pro- 
tessionals. The expected changes in professional behavior 
were seen as a natural consequence of increased under- 
standing and commitment supported perhaps by a new-found 
belief in the urgency and righteousness of the cause and 
even perhaps by an aspiration to belong to a special "com- 
munity of saints. " ^ 



By intent, there was no attempt to build an ongoing 
capacity for participants to deal with the practical problems 
of incorporating environmental education into the curriculum 
whether this involves choices at the level of the classroom 
or the school system. No consifderation was given to when, 
how and how much to introduce into the classroom or to 
suggest a curriculum should change. 

In considering any instruction there are questions of 
relevance (such as whether the exercise is appropriate) and 
style (such as how to handle sensitive issues). Some new 
teachings will succeed both in complementing and supple- 
menting the regular curriculum; some will fail on both 
accounts or perhaps fail at times but not at others. There 
will be a need to compare ideas, share experiences, con- 
solidate fikills. It is widely recognized that while some of 
this comes easily to some people, at least sometimes, other 
people need encouragement and help in getting ready for 
new. teaching responsibilities. 

This EIPIP project was not designed as a bold ex- 
tension into curriculum development and classroom trails. 
Its focus was on teacher acquisition of' knowledge and 
understanding about the environment and of commitment to 
action. Its resources would have been inadequate to such 
an extension. According to its rationale such an extension 
was considered inappropriate as a Co-op responsibility . 

However, let u s suppose that such ^yr'k/.ektension has 
been incorporated into tne ?AP\ P project , (jWh at does the 
project look likei now and^\ow might we evaluate its impact? 
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Le t us suppose > that the basic structure of the. project 
remains intact but with some changes. The Saturday ses-- 
sions feature both large group presentations and small 
group discussions . The after-school sessions focus on 
sharing of ideas and development of materials fo* in(|i;o- 
ducing environmental education in the school curriculum. 



The environmental c ontent of the Saturday morning 
sessions depends on the setting and the personnel. It has 
included topics such as: 

( a) describing and monitoring the ecosystem , 

(b) population pressures on the environment, 

(c) land use policies, 

(d) energy use and energy sources, 

(e) domestic and industrial waste disposal, 

(f) the earth^s natural resources, 

(g) water management, 

(h) food production and distribution, 

(i) pesticides, 

(j) endangered species, and 
(k) politicajl^ social/economic aspects of en- 
vironment . ' 



It also included topics of more local relevance, the 
environmental effects of a town plan, a nuclear, power 
station, a d{im construction, etc., or the complex intern- 
action of local climate , housing , resources , human needs , 
aesthetics, economics, etc. 



Emphasis has been placed on contacts between en- 
vironmental experts and school personnel^ and among the 
school personnel themselves (within and between schools) to 
generate an informal network. Information, ideas, ex- 
periences are to be shared, mostly informally and verbally. 
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Irnpact Ln the Claasroom . The Director is impressed 
with the enthusiasm and regular attendance -^of the EIPIP 
participants r but wants to know how much good they are 
getting from the sessions. One of his LTVEG interns ^ Ray 
Szoke^ is taking a course in Evaluation Methods and needs 
a project. Ray is invited to do what he can^ but advised 
that it sometimes is difficult to satisfj^ two masters, a 
professor and a Boss, at the same time. 

Ray has been favorably impressed by a report of an 
evaluation of the training of Social Security Office mana- 
gers. He takes a list of questions from that report and 
adapts them to the LTVEC situation: 

(1) Is there a sufficient number of participants from 
each school to provide - mutual support and en- 
couragement? 

(2) Do the participants represent a variety of op- 
inions about* the environment and the environ- 
mental education? Did they volunteer or are they 
obligated to attend? What sort of professional 
and personal backgrounds do they have? 

(3) Has the project been adapted to sui^ these dif- 
ferent backgi^ounds and circumstance?, and to 
make use of their special knowledge and/or ex- 
pertise? - ^ 

(4) Does attention to energy tend to ^drive away 
attention to other environment and specific ex- 
amples? 

(5) What i6 an appropriate balance between general 
knoVledge of the environment and specific 
examples? ^ ^ * 

(6) Are adequate information and /examples provided 
on each issue ^ making it relevant at all levels 
represented by the participants? 

(7) Which are valued more highly — regional or global 
issues? Are sufficient connections made between 
regional and global issues? ^ 

(8) Are the^ Saturday sessions completely and in- 
terestingly presented? What aspects of the 
Saturday sessions are most appreciated/ least 
appreciated?^ 

(9) Do participants become more know led gable about 
^ / both the global and regional environment as a 

result of the sessioni^? 
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(10) Are participants inclined to seek ^ further^ in- 
formation on their own? 

(11) Do environmental issues make participants alarmed 
and anxious? 

(1^) Are there problems in the relationship between 
Saturday and afters-school sessions? 

(13) Do the teachers become persuaded to incorporate 
aspects of environmental education into their own 
teaching? What successes and failures do they 
experience? ^ ^ 

(14) Are the sessions providing useful ideas fo r incor- 
porating environmental education into the cur- 
riculum? 

(15) Do project participants contact each other about 
environmental education or other matters? 

(16) Have the attitudes and behaviors of participants 
been changed by the project? 

(17) Is the incorporation of environmental education 
into the regular curriculum raising . issues con- 
sidered too sensitive, political, or offensive for 
classroom consideration? 



Ray's questions included questions about: 

(a) the workability of the project, 

(b) the suitability of the presentation, 

(c) the quality of interactions, 

(d) participant enjoyment, 

(e) effects on professional attitude, 

(f) effects on professional behavior. 



3uch questions are most pertinent for the project staff who 
Want both formative evaluations 

(a) to change the content and structure of the pro- 
ject, 

(b) to adjust the emphases, ^ 

(c) to improve the presentations, 

(d) to enhahce the interactions, 

(e) to provide more teacher support; 

and also summativo evaluation: 

) to decide whether it is worth replication, 

(b) to gliage the potential long-term effects, 

(c) to inform future efforts of this kind* 




Check Sheets > Ray notes that information relevant to ' 
the evaluation questions Could be sought by means of question 
naires or intervieAvs, Questionnaires 'allow more anonymity 
and are cheaper for large numbers of respondents but allow 

,onJy a limited coverage of topics. Interviews encourage 
more extensive expression of opinion ' and are more Suitable 

'for small numbers of respondents. When large scale sitatis- 
tical analysis is expected, questionnaires are almost cdways 
preferable* Ray decides to use both, a few questions for 
everyone to answer and some informal interviewing for 
probing some of the more complex questions. He considers 
eaqh-^^vas possibilities : 

(1) Preparatory Check Sheeyt — to be givern pre- 
icerably prior to the first meeting , but otherwise at the 
beginning of the first meeting , to check on the charac- 
teristics of the participants and their expectations of the^ 
project, the information would be used by the project 
directors to check whether the program is on target and to 
make any desirable modifications to make it more suited to 
the participants, 

(2) Saturday Session Check Sheet — to be given at 
the beginning of each Saturday Session to check on the 
perceptions of the previous session by the participants; the 
information would be used by the project directors to focus 
attention on desirable adjustments in the content and pre- 
sentation of the sessions and to monitor some aspects of 
their impact on the participants. 

^ (3) After-School Session Check Sheet — to be given ^ 
at the beginning of each after -school session to check on 
the perceptions of the previous session by the participants; 
the informatin would be used' by the project directors and 
the • session organizers in considering how to provide the 
most appropriate encouragement and help to teachers for 
incorporating environmental issues into the normal cur- 
riculum. 

(4) Follow-up Interview Schedule — to be used 
towards the end of the year to discover whether there has 
been or is likely to be any effect by the^ program on the^ 
professional behavior of the participants; the information 
would be most pertinent to the final report on the project 
and to the planning of similar programs in the future. 

Ray tells the Director that where any of thr^^ ir(- 
formation is' used formatively , there ought to -fcie ^Jiscussion 
of its meaning and significance at least* among the project 
directors and session leaders, and preferably too among the 
participants, before any decisions on program modification 
are made . Such disucssions might suggest the need for 
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turther informatioa and changes in the evaluation pro- 
ceduresv^ Discussion itself may produce further insight»«mnd 
understanding of the program. 

. ^/ 

In order to prepare the questionnaire and interview 
data for such discussion Ray e3<;pects to summarize the data 
and to provide a brief interpretation . There are many 
ways to do this. Ray finds that it is important to the 
Director to have a written, rteport but to discuss it " in- 
formally at a Board meeting and elsewhere^ 



Although the inservice training has been going on 

several months Ray asks the participants to complete his 

PREPARATORY CHEC^^ SHEET as they would have at the 
first session. It" is shown on the next page. 



ft 



PREPARATORY CHECK SHEET 



Date: 



This checkup is intended to help project personnel pl^|pe project 
to take* account of your expectations and needs. If ym^bink it 
dsks the wrong questions or you want to add more comments, please 
teel free to use the back of the sheet to say so. 

1. Name: . 

J. School: . .^Wfcw.-^^ 



3* Grade Levels Currently Taught: 
4, Subject Specialization (if any): 



5. Special Responsibilities ' (if any): 



b. Number of Years of Teaching Experience: 



7. What topics (if any) relating to the environment have you studied 
extensively? 



8. Wiat special areas of^ expertise (it^any) do you have in such 



topics? 



A What, kinds of experience haye you had related to environmental 



iss 



10. What kinds of environmental concepts <if any) have you recently 
• taught? 



It. What is you most important reason for participating in this 
project? 



12. What special concepts and/or skills do you hope to learn from 
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Ray wants to - get some information from the par- 
ticipants while a session is still fresh in their minds. 
Realizing that not much time should be taken from dis- 
cussion, and in order to plot responses from session to 
sessionm that some of the same questions should be used 
each seesion, he shortens and combines the Saturday and 
After-School instruments into the SINGLE SESSION CHECK 
SHEET. He decides against anonymity (readizing that 
asking respondents to identify -themselves will prompt a few 
tu respond in a way that reflects well on themselves) but 
he wants to relate these check sheet response^ to par- 
ticipant characteristics obtained on the PREPARATORY 
SHEET. 

Ray administers the sheets himseff and reminds the 
participants that they are under no obligation to respond, 
but that it is an impoirtant favor to him if they do, and the 
information may make fiUture sessions better • 

/ 

The testing goes well. As usual a few resf)ondents 
tind tht^ scale ter^s ambiguous or overly simplistic ^nd 
write marginal comment T he comments do not indicate 
hostility as much as an inability to say in the scaling 
medium what they feel. . 

But Ray had attended tiiose sessions and he was 
disappointed b«*cause the generality of tne items, as wi 
most attitude scales, did not deal directUy with the topics 
and incidents of/ importance • He reread thVonening chapter 
ut Shaw and Wright's book, then decidecN-io prepare an 
end-of-term seeing sheet that did deal more directly with 
what happened in thvj sessions. titled this scale, A 

SUMMARY OF OPINIONS ON ElPIP. 
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• -'SINGLE SES?^ION CHECK SHEET 
N'ime: ■ Date: tt' 



The tollowing questions are intended to obtain a quick 
summary ot your jmpressi<5^s of the" last session. 



Pleatjo place a check ^ mark (X) to represemt your, opinion 
along each scale. 



I. Mow important was the topic for you? 



extremely ' extremely 

unimportant neutr^il important 



How helpful was the session to a teacher trying to teach more about 
the environment? 



xtremely * ' somewhat helpful " extremely 

unhelpful somewhat unhelpful helpful 



I. How interesting was the session? 



extremely , neither, or 3ome extremely 



boring • oi-O^qth 



interesting 



4. It the session were repeated, how would you want to see it changed? 



5. Have yoKi any other comments, criticisms > suggestions? 



8:, 
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Your 
Name 



A SUMMARY 01- OPINIONS ON E IP IP 

To indicate yoff o[)Lnions of the HIPIP project please 
_^ mark an X at the a[)propriate place on each scale. 



1. Was there euouyh opportiuuty for discuss ion 

la) at the Saturday sessions'^^' 
(b) at the atter-school sessions? 



Were discussions lietter because participants 
came from different school districts? 



Are you now more aware of environmental issues 
m your daily life than you were before UlPl-.P? 



much- too 
1 itt le 



much 
better 



-L 



about 
right 



no 

d if ferent 



J. 



much too 
much 



much 
worse 



not at all \ some 



verv mucl^ 



I. Was tlie [u-oitjct valuable to you personally? 
3. i)id the Su[)er m tendent s dominate discussions? 



not at all "^ome verv much 



not at all 



lias your hi PIP Participation led vou to discuss , 
environmental issues with uon-part icipat ing educators? never 



some 



some 



verv much 



many times 



W;is the P.IPIP discussiQn of mass transit and 
new ex[)ressways valuable to you? 



not at all 



some 



verv much 



S. Why was tl^M'e \o more discussion 
on tlie Tellico Pam issue in your, yroup' 
((Iheclv more tlian one if your wish) 



(a) too bor ing 

(b) stilLtgo sensitive 

(c) (othbr) ^ 



Please indicate junv relevant each of the foIU)wing 
topics (from the Saturday sessions) were, in your 
•n 1 nd , to the use of local environmental issues in 
teach ins^t 



10. 
'l 1 . 

^13. 



Describinv^, and monitoring, the ecosystem 
Population [)ressur&s on the environment 
l-ners;y use and enerp^y sources 
Domestic and indu.^trial waste disposal 
Pood product i,on and distribution 



11. 
"l&. 

^1(K 

^18. 



0 ^ almost unrelated 

1 =» a little related 

J = fairly well related 

3 - extremely related 



land use po I i c ies 
The earth ' s natural 
Water management a 
Pest icicles Ay 
lindangered S[)ec ies 



resources 



19. What did vou like most about filPIP? 




20. What did you dislike most about lilPIP? 
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Ray atirninistt^reci this scale and showed the results to 
the Kxecutive Director. They found several thinj^s to 
puzzle about> particularly the reserve the teachers had* 
about being identified with environmental advocacy. Ho 
asked Ray if' he could find out more about it • Ray called 
Claire Brown at CIRCE (Z17 333-3770), knowing she is a 
specialist in program evaluation who had done a case st;udy 
of LTVEC, Claire suggested that he might try an attitude 
sclae developed by Graham Maxwell and B(^b Stake > a scale 
called the RELS SELF- ASSESSMENT SCALE. (A, copy is 
included*) No technical information was available for this 
scale, but Ray felt that he could get a better picture^ of 
how the participants now feel about a commitment to en- 
vironmentalism . 

Ray wished he had administered the form as a pretest 
as well as a post -test • This would have given him a gain 
score for each participant, and- he could have tested the 
statistical significance of the gain on each item. He was 
aw,>H-^ that gain scores ^e much less reliable than the scale 
scores . 

The median score for most of the knowledge items on 
She RELS SJLLF ASSESSMENT SCALE indicated that the 
particii^ar^^^ fel t they knew more than "a small amount" but 
less than large amount" about various information and 
issues. Only 2 or 3 participants indicated a strongly 
activist stance, and most were optimistic about the benefits 
of collective inquiry and, curricular attention to en^ 
vironmental problems. 

Ray did not do any hypothesis testing of the statistics 
because he was interested in describing this particular 
group rather than drawing inferences to some hypothetical 
population. 



Summary. Ron Szoke made a rough summary of data 
from each of the \ four opinionnaires and spent a couple of 
hours discussing them with the Director. They went over 
fs'ich of the original 17 questions, noting no data on many 
of them but concluding that t^here were good answers to 
rnanV/- questions about particip-ant interest, conviction, and 
llk(3^ihood of taking further action. He asked Rgn to make a' 
brief oral presentation to the BoSird , which provoked a 
lively discussion for almost an hour one evening. 
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RELi^ SELF^ASSESSMENT SCHEDULE 

Please indicate tjie amount of each characteristic or value true for you. 

0 = None or almost none 1 = A small amount 3 = A large amount 

(Let'^ let 6 = the amount for a Ralph Nader or Albert ^:instein 
or the most extreme person on any particular scale) 

Write the appropriate scale value (0,1,2,3,4,5, or 6) 
for you in the space for each item. 



1- YoXir knowledge of general environmental issues. 

2. Your knowledge of environmental issues of this region. 

3. Your knowledge of the ecology of this region. 

4. Your knowledge of agencies for environmental control. 

■ 5. Your 'understanding of the interaction of environment, energy, and 

economics. 

t 6. Your understanding of social and technical obstacles to environ- 
mental control. 

: 7. Your interest in working for environmental protection and improvement. 

Your sense of urgency about environmental problems. 

^- Your sense of despair about resolving environmental problems. 

10. The importance you attach to collective inquiry for resolving 

environmental issues. 

11- The possibility of resolving these issu^ by community discussion 

and debate. 

12. The need for aggressive protest against those causing environmental 

problems. 

. 13. Your ability to contribute to public discussion on environmental 

issues. 

14. Your .frustration with people having little interest in environmental 

issues. 

15. Your knowledge of potential sources of environmental informatiorK 

16. The possibility of adding environmental education to the, school 

curriculum. 

_____ 1/. The value of adding environmental education to the school curriculum. 

18. Your intention to work for adding environmental education to the 

curri culum. 

19. The need for calm and rational study of environmental problems. 

Your sense of commitment to -envi ronmental education. 

ERIC 
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Illustrative Application #3- 

AiN KVALUATION STUDY OF POSSIBLE 
BIAS IN COMMITTEE WORK * 

Focus: Citizens League of Minneapolis-St . Paul 



Evaluation studieg are often undertaken because some- 
one complains about the way things are working.. One 
common complaint about environmental study groups is that 
they are biased in one way or another. It is expected that 
individual members of committees wiU have their convictions 
(biases) but that the committee as a' whole will have a 
balanced approach and that 'its reports will not overly 
emphasize or ignore the viewpoint of any special interest. 

From time to time critics of the Citizens League have 
charged it witj;i "elitism'^ and with '\under-representing 
women^ minoryties^ and low income groups," With every 
committee report there are some who feel that recommen-*- 
dations went too far or not far enough, and thus were 
biased. 



V In Chapter II readers were informed of League manage- 
ment efforts to selectively recruit additional committee 
membera ato balance sides out, During orientation sessions 
invited 'speakers acquaint the committee with "the facts of 
the issiies under consideration." Many valued resource 
persons represent special interests • The staff member 
serving the committee attempts to assure, not that bias will 
be avoided, but that ail sides are heard. ^ 



Of course a well informed commi.ttee Jfwill never be 
entirely unbiased. And iJiaanimity doeS^ not guarantee 
fairness. Experience has shown that toward the end of 
League study sessions cpmmittee members are usually 
largely in. agreemeLnt over the key issues and some action to 
be takein-. This is true no doubt partly because many of 
thoj^e seeing little hope for their point of view drop out. 
And thos^ stayiqg gravitate toward a compromise position > 
not necessarily fr'ee of bias but minimally controversial 
within the committee . There is no hope for a report en- 
tirely free of bia$> but there are many ways of working 
toward minimization of bias. Some were just mentioned. 
Another way is for someone to evaluate the process to 
identify sources o( bias. That is the so^t of evaluation 
study described in this section. % 
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Self Stucl y. For purposes of making this evaluation 
manual realistic let us suppose that about four months into 
Its work the Study Committee investigating , the topic 
SELECTING SITES FOR UNDESIRABLE ENVIRiONMENT AL 
USES has opted to conduct such an evaluation. Members 
were concerned about possible bia3 in their own delibera- 
tions and had to wait several weeks anyway until further 
testimony was available. 



The committee considered its resources and its con- 
cerns carefully and decided that the committee members 
themselves should do the evaluation, ;rhat is, they chose 
to do a "self-study." The resources would largely be those 
already at the committee's disposal and the informatign to be 
examined would largely be that already shared by the 
committee members. Still, members are aware that the 
process of self-study is frequently criticized for ignoring 
the values of outsiders--an allegation even more relevant 
.for the study of controversial issues. Therefore, its 
self-study is planned to use both internal and external 
procedures. 



The committee does not intend to study Citizens 
League operations in general but feels that study of its own 
operations will be useful to other committees and to the 
League's Board. Therefore it accepts the responsibility for 
making certain matters more explicit, detailed, and read- 
able. And to protect the confidentiality of certain matters, 
it need be, by presenting some testimony anonymously and 
possibly with some obfuscation of personal circumstances. 



V 
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The principal question is: 

What evidence is there that th^ ^com- 
mittee does not give balanced attention 
to all sides of questions and that 
materials for the final report are inade- 
quately considerate of certain important 
value positions? 

The members of the committee decide^^they will break their 
work into four parts: 

(1 ) Identification of both general value 
concerns and concerns specific to 
an individual issue 

*\ 

(2) Review of committof^ action by 
persons sympathetic to the indivi- 
dual value positions 
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(3) Compilation of instances of possible 
bias by committee members 

( 4 ) A nalysis and presentation of evi- 
* I dence of bias 

^The committ\?e divides itself into four subgroups for those 
tour responsibilities, with the expectation that other people 
will help when needed. They decide that they wiU try to 
complete this evaluation within a month, using parts or all 
t. of the four evening meetings regularly scheduled for this 
evaluation work. 



Value Position Acknowledgment . The first subgroup is 
scheduled to bting to the first meeting a list of value posi- 
tions important in their work . They discuss the situation 
and decide to include three elements in their list: 

(a) value-laden issues the committee is 
con^sidering 

(b) specific^ advocacy groups indicating 
concern about these issues 

(c) general groups of people having a 
stake in the issues 

They each agree to work on the list during the coming 
week so as to present a list for discussion and improvement 
at the first evaluation meeting. One member agrees to 
recced, by phone, the suggestions of subgroup members 
and to present a list to the full committee at the first 
evaluation meeting. 



At that meeting a list of 12 issues, 21 »fedvocate 
groups, and 9 general groups is discussed, and added to. 
The committee becomes persuaded that they should try to 
work most with a smaller number of issues and groups, 
checking, though, to see that the substantial range is 
maintained. Their Qoncern was not to identify all possible 
value positions, and thus all possible sources of bias, but 
to identify a» small but diverse working set of value posi- 
tions for thorough examination. The resulting list was 
turned over to the whole committee: 
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REPRESENTATrVE VALUE-LADEN LSSUES BEFORE THE COMMITTEE 
(this list and" the entire story are hypothetical) 

(I) Location of Solid Waste Disposal 

Impact on : taxpayers, nearby residents,, land developers, junk 
' dealers,, citizens 

Petitioners Associated Refuse Haulers, municipalities, Save Our 
Cities, American Federation of Labor, individuals 

(J) The Disposition of Nuclear Plant Wastes 

Impact on : citizens, power and energy *users , residents 
^Petitioners : Northern Electric Power Co. , Minnesota Chamber o^ 
Commerce, Sierra Club, American Civil Libertie^ 
Union, individuals 

/ ^ 

(3) Industries Depending on .Natural Water Cooling . 

Impact on : citizens, taxpayers, riverside residents and busi- 
nesses, sports fishermen 

Petitioners : Isaac Walton League, Northern Electric Power Co., 
Ten Thousand Lakes Tourist Association, Outdoorsmen 
Inc., Sierra Club, Inland Smelting, individuals. Uni- 
versity of Minnesota 

(4) Cattle Feeder Lots 

Impact on : citizens, taxpayers, nearby residents, land devel- 
opers, cattle feeders, meat packers 
. Petitioners : Swift & Co., General Foods, American Farm Bureau, 

• ' , municipalities, American Hereford Association, indivi- 

duals 

(5) ZonxnfT for Power Boats, Snowmobiles and Other Noisy Sports Craft 

Impact on : citizens, taxpayers, sportsmen, land developers, 
craft retailers 

• P(?titioners : Outdoorsmen Inc. , American Red Cross, Minnesota 

Chamber of Commerce, Sierra Club, individuals 



This list was accompanied by a statement indicating that 
these issues And parties could be considered representative 
of committee involvements for the purpose of deciding 
whether or not certain rights, investments, and ideals 
might be und^r-^represented in actionlis taken by the com- 
mittee. 

J 



9. 



83 



A seco/id subgroup took the responsibility to arrange a 
review ot committee action to date by persons sympathetic 
to inciividual value positions. After looking at the list 
presented by the first subgroup they decided to ask the 
following persons to meet at the regular meeting time of the 
second evaluation week: 

(a) an executive of Northern Electric 
Power 

(b) arj^ officer of the Sierra Club , 

(c) a staff member of th« Chambef of 
Commerce 

(d) a Legal Aid attorney 

(e) a resident living near a challenged 
industrial site * ^ 

A League staff member (other than the one regularly 
assigned to this committee) agreed to assist this panel. 



The primary materials for tH^ir review were the weekly 
minutes , noted in the previous chapter as "terse , cogent, 
completA and about as interesting as minutes can get." 
Various attachments jand exhibits came with the minutes. 
The charge to/ the '"blue rilibon panel" was as follows: 

Please examine the materials of the 
' * committee studying SELECTING SITES 
FOR UNDESIRABLE ENVIRONMENTAL 
USES anc( note any evidence you might 
find gr any suspicion you might have 
^ that the deliberations of the committee 

are incomplete or biased toward or 
against any point of view. Note any 
reasons that might occur to you why the 
committee might be so disposed. 

The panel received their materials the day after the,, first 
evaluation meeting aVid agreed to look them over before 
gathe^ring at the end Kfthe wjj^k. The panel meeting would 
be used for discussion ^ancl further ^tudy . The panel^s 
staff member agreed to have a summary of their findings 
available for distribution at the third evaluation m:eeting. 



At the suggestion of one of the committee members the? 
panel e.»f forts wer« to follow a "goal-free evaluation" 
approach as advocated by Michael Scriven. Although they 
might be somewhat familiar with the goals of the Leagire and 
this particular committee, ^nd although they would 
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encounter some goal statements 0\ the minutes, they were 
not encouraged to conduct their investigation with parti-.^ 
cular attention to those goals, but to look only tor evidence 
ot bias as it was meaningful to them. 



Each member of the panel had comments to make about 
how special interests were being handled. There was some 
general agreement that the committee, was responsive to 
advocates and petitioners, but noted that the choice of 
testimony and spokespeople was, as recognized, incomplete 
jind somewhat arbitrary. Further they noted that the com- 
mittee seemed to attend more to points of view presented 
personally, less to those well documented but not presented 
personally. ^Two panelist^ separately lamented particular 
inattentions, one to the concerns of the taxpayer and the 
<^her to the need for a vigorous and expanding economy for 
full employment and for sol vency of the social security 
program. In other words, to the panel, committee biases 
seemed determined more* by their own processes of inquiry 
than by prior convictions of • the committee. 



Self-evaluation Portfolios . The third subcommittee had 
the responsibility to compile instances of possible bias rec- 
ognized by members of the committee. At the first session 
the subgroup discussed the use of self-evaluation port- 
folios. Following the work of. Scheyer and Stake the plan 
woidd be for each member to keep some reminder (in a 
portfolio) of each instance when the question of bias arose 
but did not get immediate attention. Periodically the port- 
folios would be sorted out for setting agendas aimed at 
discussing the control of bias. Since this evaluation effort 
was to cover activities already past and was to result in a 
summative rather than an ongoing review, the subgroup 
asked each committee member to pretend that they had kept 
such a portfolio and to think of two or three items that' 
might be in it, and to prej^artT a 3x5 card briefly explaining 
the instance of bias referred to. If it was an instance not 
known to most members of the committee the explanation 
;rnight have to run a little longer. These 3x5 cards would 
/be .collected after some review anc[ discussion at the second 
mee'Ung and prepared by, the League staff member for 
distribution at the third meeting. 



L give the committee members more of an idea of the 

rationale and procedure of a self-evaluation using portfolios 
the 5tkff assistant circulated a brief explanation, including 
the following quotations from Scheyer and Stake: 

The portfolio shoidd include the judg- 
' ments and opinions of various persons. . . 
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J On each issue there will ^e different 
value positions • Diverse tesitimonies 
reveal the different perceptions that 
people have about what is good and bad 
about a program. • . 

As different valuings become clearer, the 
need and ^the opportunity for compromise 
emerge • The^^-e will be times ^he staff 
should resist compromise, to hold out for 
what is * precious to them; but Usually 
some compromise should^^be found to 

respond to concern from all side^, 

< 

Again attention was drawn to the importance of recognizing 
imp^^dt bias : 

• the portfolio not merely a 

happenstance collection . Some entries 
are sought and others are created for a 
purpiose. Some things need to be repre- 
sented more than others* . . 

/ 

(This) portfolicin should be a loose collec- 
tion so that ^ parts of it can be 
rearranged and differently displayed 
from^^ime to time* The entries should 
reflect th^ program activities, its issues, 
its valuings, and its compromises. 

At any one time the ^ portfolio keeper 
presents only a selected portrayal. The 
selection should be based upon issues, 
key issues, major concerns about 
running the program. Only a few issues 
should be pursued , but thgse in 
depth, . • 

Progressive focussing on the evidence in 
the portfolio sha>pens the portrayal, A 
store of responsive, expressive questions 
about the program begins to emerge. 
The questions represent the issues, . . 

At the third meeting the committee found the following 
among the many noteworthy items in the simulated port- 
folios : 

(a) an instance when the committee 
chairman urge?d that the delibera- 
tions put personal concerns ahead 
of Social concerns and social con-- 
cerns ahead of commercial concerns * 
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(b) the d<»cision ot the committee not to 
consider parking garages and 
limited access highway location 
withirj. its jurisdiction 

(c) the unwillingness of the committee 
to hear the full testimony of a 
woman who wanted a clear distinc- 
tion between nuisances and hazards 



An Adversary Heanng . The fourth group decidedNit 
would like to have the evidence of bias brought to the com- 
mittee as if it were a Congressional hearing, a program 
evaluation approach advocated by Owens, Wolf and others. 
At first there was opposition that would tend to politicize 
their sincere efforts— but a 1968 quotation from Congress- 
man Boiling brought agreement^n^erhaps relugtant: 

I would hope that All of you who are 
disillusioned by ^t]/4* political process, 
C who think you can escape the political 

process, would recognize that the frame-. 
work of the society withia which you 
work and plan is based on the political 
process. 

Two attorneys servirtg on the committee were persuaded to 
summarize and present the evidence of bias and the refuta- 
tion of that evidence, but only after it was agreed— as Wolf 
urges~-that both sides would be pledged to the clarification' 
of the situation rather than to the winning of the confron- 
tation. In fact, each advocate agreed to commend his 
opponent when he felt that a point had been particularly 
well made. 



The remaining members of the Aibgroup all decided to 
sit as hearing officers and the rest of the committee would 
sit as audience. A third attorney volunteered to serve as 
hearing officer or moderator. The principal testimony to be 
heard would be that submitted by the first three sub- 
groups, the value positions list, the findings of the blue 
ribbon panel, and the items from the simulated portfolios. 
Each side was ad lowed any additional witnesses it needed 
but each of the two presentations, not Including questions 
from the hearing officers and cross examination/ was not to 
exceed 40 minutes. At the close of the: hearing the entire 
committee would vote on the following motion: 

Because the actions of this committee arc- 
in need of more effejctive ^ ^control of 
possible bias, a subcommittee will be 



named to devise- and initiate various 
remedies^ subject to approval of the 
committee as a whole* 



f 

Interviews on Representation , One member of the 
committee continued to argjue that bias could best be under- 
stood and controlled^ by increasing attention to the selection 
process. She Cirged that another evaluation study be 
undertaken, perhaps by another study* group, with inter- 
vi^'^v data from five grojiips:* 

(a) the Leaguers ^ Program Committee 

(b) original and persisting members of 
the study group 

(c) .othe^* persisting members added 
later 

(d) study group members who dropped 
out 

0 {e\ consultants, petitioners and other 
resource persons 

She proposed that the interviewers organize their ques- 
tioning around these questions: 

(a) What do yo'u see as the* basic pur- 
poses of the Citizens League? 

(b) What do you see as the basic pur- 
poses of this Study committee? - 

• (c) What at^e your personal interests in * 
the work of this study committee ? 

(d) What are the activities you have 
been most, pleased with? 

(e) What are the activities' you have 
been most dismayed about? 

. (f) Do you feel that in any instance 
the committee has been unfair? 

(g) How could the committee improve its ' 
representation of the various points 
^ of ' view and arrive at fair and 
effective solutions to problems? 
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She felt tKat the League .4houId underwrite the costs of 20 
days ot interviewing plus analysis by an experienced and 
respected interviewer. 

Other committee members commented informally that the 
one place to study bias of the committee would be to study 
the role of the staff member. The data are "filtered"' by 
the keepers of the minutes. Many subjective choices are 
made. • 



An examin^ion of the procedures by which committee 
members are selected, of the quality of services obtained 
from consultants, and of* the operating procedures of the 
committee itself,^ is part, of the responsibility of each study 
committee and the parent body. To some extent watchful 
eyes are ever watching: officers, ' newspaper people, 
citizens— friends and enemies. But is it enough' to monitor 
the work informally? Often, yes; sometimes, no. 



Quality Control of Network Information . Obviously, 
the combination ot goal-free evaluation, external" review, 
portfolios and an adversarial method of issue resolution 
places a high premium on "the integwty and fairmindedness 
of study committee members. Even with fhe resources of 
the Citizens League and even with the services of excellent 
staff personnel, the resources of the individual member are 
greatly needed. 

1 

\ 

Research on the "transfer of knowledge" supports the 
quality-control steps emphasized here, including the careful 
balancing of committee ^membership, utiliziiSg concultants. 
and gathering information about the study process^. Menzel 
and Havelock testified to the importance of such steps. 
And a summary document on the factors involved in the 
successful transfer of innovations said: 

An outi^ide person or group may perceive 
a need for new knowledge and also take ^ 
on the responsibility of bringing the * 
knowledge to the organization. 

Menzel extended this premise into the realm of networking , 
a concept prominent in literature on environmental education 
rind central to the case studies presented in this evaluation 
manual^ Menzel said; 

Many new information networks have 

been established by organizations such ^ 

as the Council of State Governments. . . 
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By organizing conf^er^tes of publishing 
newsletters they iM'in^ together officials 
from all over the (jj^i^untry and facilitate 
the exchange ^f ^ ideas and knowledge, 
thus increasyig the officials' awtreness ' 
of the'iates^de^elopments in their field. 

In some importa/t 'Oi^ays, the emerging ' n^etworks are a 
reaction to traditional organizations which^fat least in the 
eyes of critic^-a^e not responding efle-CtiVely to changing 
conditions. 



In £om^ places "networking" 'is no more than a buzz 
word, ^tr0ng woi^'ds of caution about widespread dissemina- 
tioiy'di sniperficial ind rationalistic are to be found in the 
MinijfBapoliS'-St, Paul material and other "information science" 
doQiuinents, Given the political-social-economic trade-offs in 
tb)e fields of energy and environment today ^ warniYigs by an 
iir;6ht scholar should be heeded. In 197S Everett Rogers 
^advise'd that: 

shedding the pro-innovation bias in 
diffusion research should lead tq serious 
questioning of how 'good' 'the innovations 
being investigated really are, rather 
than assuming that tlxey have consider- 
able relative advantage over previous 
practice. 



Awareness, Knowledge and Behavior , Beginning with 
the ESEA legislation of 1965 , and accompanied by other 
activities initiated by' the late President Johnson's plans for 
a "Great Society," we have received a host of reports on 
the processes and procedures by which awareness leads 
individuals and groups through a series of stages to an 
end-result manifest in practice. Usually, this alledged 
sequence moves from awareness — to acquisition of knowledge 
. — to trials or comparisons of the new infornhation with older 
forms — to adoption of the behavior or innovation. It has 
become Increasingly evident that few, if anySsodal chang^ 
actually occur in this linear fashion, Lindbloom's famouW-. 
critique of 1959, and his praise for "muddling through," 
has been followed by countless other criticisms of * tech- 
nological advocacies. Pages 7 and 8 of the Minnesota^ case 
study highlight events that fit poorly into linear models of 
change,' ^ 



First 



they cite the critical influence of 
visibly problems — e . g , , evidence 
oi Dutch Elm Disease* 



.9. 



90 



Second 



events in Syracuse, New York, 
are hailed as testimony to the 
need for sustaining interest and 
support over time* 



Fourth 



Third : while awareness may be acute, 
the necessary knowledge may not 
be. available — ^or may be controlled 
by thos'fe with vested interests in 
keeping, it 'out of the public 
domain* Also, the high degree of 
^ emotionalism surrounding environ- 

mental issues is already apparent, 

we live in a knowledge-^rich 
society. Not all of it ^is equally 
valuable , but it all competes for 
our time and attention • The 
existence of such conditions 
reinforces constraints — latent and 
real — encompassed by the pre- 
ceding trio, 

I ■ 

At least one other matter looms over our efforts to 
Gope with environmental issues in an "educational" format. 
In the, form of a question it might be presented this Way: 
"Will conflicts over the environment reinforce social class 
divisiveness?" This concern was expressed in our opening 
statements ♦ Intervening passages have attempted to 
delineate certain evaluation measures designed to reduce the 
controversies surrounding environmental problems. But 
contlrdvetsy will remain. ^ 



J 



t ■ 

i'^nal vticaKskilli 



Traditional forms'^of evaluative and'^inalytical^s kills may 
have to be supplemented by the tools of conflict resolution: 
negotiation, bargaining, compron]ise, cioriciliation , creative 
integration and synthesis, organization of coalitions, and 
techniques^ of individual and group persuasion. Burden^ 
and responsibilities assumed by members of the Citizens 
League and its cadre of study committees seem almost over- 
powering. Yet they appear better equipped than most civic 
groups to confront the major challenges of our time . In 
fact, environmental struggles may offer the severest test of 
the final criterion employed by the Program Committee in its 
efforts to help League members decide on projects — namely, 
'*Is the problem capable of being resolved by reasons bksed 
on fact, ^r are the emotional overtones , the tides of /bias, 
too strong to pennit reasoned analysis?" 
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Illustrative Application #4 

AN EVALUATION OF STUDENT ACHIEVEN^ENT 
Focus: Little Tennessee Valley Schools 



Perhaps the most cortimon type of formal 'evaluation of 
educational programs is the evaluation of student achieve- 
ment. It is common to think that the quality of the teach- 
ing is manifest in the quality of student performance on 
tests. Actually, the major determinants .of student per- 
fgrmance are previous achievement and aptitude rather than 
teaching of fectiveness , and our tests are too crude to give^ 
a good picture oi, what youngsters have learned. Still, 
achievement ' tests sometimes give us at least a rough idea of 
what the teaching has accomplished, « ^ 

'A 

Unless the \est is standardized and we are very 
familiar with'' usages of that test, the total score tells little 
about the attainment of the student. (And standardized 
tests are seldom closely related to the teaching,) 
Individual it^ms may tell us important things. By 
answering correctly, does the student krxi>Ar this particular 
fact or relationship alohe, or the surrounding complex of 
facts and relationships. We know that our items fail to tap 
many of a youngster's understandings, so that it is easy 
for scores to underestimate attainment. But it is easy for 
us to ovtergeneralize from a correct resppnse, Yet\ even 
that limited information can be valuable. 

Teaching Envirc/nmentalism Let us assume that four 
teachers 'attending the EIPIP ins^rvice session decide t6 try 
out some environmental education materials in their own 
classes ♦ They decide they will particularly try to under- 
s tan/1 the difficulties of changing course objectives under 
pre^nt school circumstancesl^ ^ 

^ The teachers and administrators at the Saturday morn- 

ing sessions frequently pondered the issue of how more 
environmental questions could be raised in school. They 
wer'e sensitive to the fact that the public and the profession 
were pressing for less^ to be included in the curriculum, not 
more. Most were persuaded that the curriculum reform 
efforts of the 1960s had raiseti too high the expectations gf 
what a teacher would teach. The watchword was "back to 
the basics," New courses ^and new content within the tradi- 
tional courses had few advocates. 
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Yet they felt tH^t the perplexing problems of energy 
^ind environment should not be left entirely to incidental 
learning. Television portrayed some things effectively, but 
the personal involvenyent needed for a real g-rasp of the 
social, political, economic and personal interrelationships 
was missing , ' ^ , 



The best scholastic solution seemed to be 'to provide 
supplementary units to teachers who would deci,de, on an 
individual bdsis, when there was a need and an opportunity 
for teaching basic environmental understandings. ' 

I 

The four teachers who decided, on som^ immediate 
try-outs are: 

Frances Winters, social studi6^, CoUingwood 
School 

Rose Jackson, home economics, East iVtonroe High 
School 

Forrest Hagerstrom, mathematics, East Monroe 

High ^ . 

Shirley Webb, business education, Cherokee 

Unified ' . 

Eacli of the four picked outgone course s/he was teaching 
and an environmental topic that seemed to fit: 

Sociology . . . • • .Energy 

Marriage and Famil'y" , , Population Growth 

Practical Math Utility Bills 

Consumer Problems. . . Conventional vs Microwave 

Ovens 

Shirley Webb indicated that in her Business Education 
course she wanted to introduce the topic of "demand for 
electric energy" but needed Something like "the choice of 
ovens to purchase" as a practical point of entry. r 



These teachers were developing many"\of their ideas 
from materials published by the University ^f Tennessee 
Environment Center • Franpes Winters found, for example, 
ideas in their "white book" /or high school social -studies on 
t1>e following : 

(1) Socioeconomic Problem^ and Energy 

(2) Population Growth and Resource ^ Consumption 
, v3) U.S. Dependence on Foreign Oil 

(4) Energy Production and Wasted 

(5) Rntail Merchants and Energy Conservation 

(6) An Energy Conservation Campaign 
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Forrest fl^^^rstrom was in correspondence with 
Professor Ro^'rt Waller of Mothern Iowa University to 
obtain mathr exercises in an environniental education 
context, ^ , ^ * 

OveJ a period of two months each ^teacher \vas 
e to apend at least three hours of class time on the 
nvironmental topics. At first they felt that this was 
too little to produce any measureable achievement, but 
i,hey were curious about that, and they wanted to get 
a feel about handling it as a full teaching module. 
They decided to go ahead with some testing. 



Rose Jackson had heard that criterion-referenced 
tests were superior to the usual norm-referenced tests 
for assessing performance on specific objectives. None 
of them really understood what that meant so they 
placed a call to the ERIC Clearinghouse on Tests and 
MeasuVements (609 921-9000). They were referred to 
several books but also were told that for constructing 
a few items for their classes it was not going to fn^ke 
any difference \Vhether thisy wrote criterion referenced 
itpms emphasizing skills that all students^ were ex- 
pected to master ^ or whether they wrote 'norm refer-^ 
enced items that were intended to discriminate max- 
imally between the students who had acquired the 
general skills and the students who had not. 

Item Selection . They decided it would be in- 
teresting to present some o^ the same questions to all 
four classes, plus additional questions more pertinent 
to their special lessons. For the common questions 
they selected True-False questions from the University' 
of Tennessee Environmental Center^ materials. The 
•twelve items selected are shown on the next page. 
For assistance in constructing other items they re- 
ferred to Gronlund^s book. 



After each teacher had administered the common 
and' unique'^ items to their class they got together to 
discuss the results. They concluded that the scores 
did, as usual, give them a pretty good rank-ordering 
of students from best achieving to poorest, anS that 
more students missed several pimple items than they 
had predicted. But they also knew of several^ stu- 
dents who had really gotten involved in Ujie environ- 
mental questions, had read several relevJ^t articles, 
yet had done poorly on the tests. Generally^ th^y were 
disappointed with what the test results told them, anc 
felt it would have been more useful to have had ^a 
discussion with some of the students to find out what 
was understood and how the instruction could be 
♦improved. 
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T-F ITEMS CHOSEN BY EIPIP TEACHERS TO USE IN COM- 
MON FOR TESTINQ ACHIEVEMENT 

(1) The population of the United States doubles about 
every 46 years and the demand for electr^^city 
doubles about every 10 years, . 

(2) Widespredp conversation of urban refuse into 
fuels coiid soipply approximately 25 percent of 
our total Wniiml energy consunip^on. 

(3) If all coal from- reserves in IShe U.S. were mined 
and made available, there would be enough to last 
us for several hundred years if used at the 
present rate ,of consumption. 

(4) ^The cost of' transporting coal can often equal the 
cost of mining . 

4 

(5) Oil'is the cleanest burning of the fossil fuaJs* and 
is, therefore, in great demand>s^ 

(6) The most) significant variable affecting fuel con- 
^sumption in an automobile is its weight. 

(7) Th4 best location for the thermostat is on the 
boldest wall. 

» * 

(8) Fluorescent lights are less efficient than in- 
candescent; 

(9) The energy used to drive a -car initially came 
from the sum via green plants. * , 

(10) The heat lost through the house is primarily 
^through the ceiling. 

(11) In the Tennessee Valley region, the water heater 
accounts for about 50 percent of the electric bill./ 

(12) The electric range uses less electricity ' thati the 
'clothes dryer, window, air ^ conditioner^, aijd 

dishwasher combined. 
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Illustrative Application #5 . - . 

ETHNOGRAPHIC EVALUATION OF A COMMUNItY-B ASED 
PRg^GRAM ^ 

Focus: ';^>h,^ Land Owners Organization . / 



Consider the following description from the notebook of 
thf^ LTVEC case study ethnographer: 



^*In ' a day of considerable government talk, about . ad-- 
vocacy ^nd erapbYment of the people, we tsee that in an 
indirect and painful way, TV A has empowered the people of 
the Little Tennessee^ Valley as they "would not have been 
empowered otherwise. From experience with TVA and the 
Tellico Project^ th^ people have learned abc^ut the im- ^ 
portance of community and regional advocacy. They^ve 
learrted, vrith TVA^s good example, the meaning of the 
words from' the. popular country and western song, '^The^ 
Gambler," so big in Tennessee this spring; 

You gotta kno^v when to hold ^m 
Know when to fold 'em 
now when to walk awav 
^ Know when to run. ... ^ 

''This is not to say that in the post^ellioo period 
Loudon, Blount, and Monroe counties resonate with com- 
munity action. Still, there are a 'few indication's of a^ n^w 
.way of thinking about issues related to the Little Tenness)^e 
and the river valley, and surprisingly, of -support for that- 
activity from TVA^ 

"A gctod example o£ \this more region-'oriented , 
action-oriented approach*, as ^ell as TVA's new posture is 
seen in the Little Tennessee Land Owners Organization. 
Times were particularly bad in the days w(hen people along 
the Little Tennessee were having their homes and land 
snatched rather ruthlessly from them, 'with token reirrj- 
bursement made by TVAT Quite predictably, there evolved ^ 
a looselyknit very informal lietwork of displaced and po- 
tentially displaced land owner's — 603 in all— neighbors be- 
fore, now linked together even mor*e^ closdly by their com- 
mon problem. Through the network flowed mutual support, 
and information such as which land TVA was appraising 
now, what was being offered, -what suits were being filed, 
and so. on. Of this group, five families refused to move at 
TVA^s demand (and at this writing still have not), among 
them the R^ichies, the McC^ls, and the^Mosiers. The Davis 
family sold, but solicited the lej|al services of a special 
lawyer from Kentucky, and filed suit against TVA. 

10^ 



. "In the fall of 1978, the new director of TV A, David 
Freeman, encouraged the structuring of "this network into* a 
more formal orgahizatiqnN /According tb Al Davis , current 
president of the Land Owners Organi-zation , the organization 
has a public membership, elected officers; and meets twice 
monthly with two stated purposes: 

* ^ • 

(1) to *work toward regaining the land for the 
original land owners. 

( 2 ) to work to protect the rights of the land 
owners • 

V 

The LO A formulates and states positions on issties such as 
the leasing of the land, now owned by TVA, and protection 
from poachers < on the^land. * PositioK statements are written 
and submitted to TVA and TVA staff meet with LOA officers 
to discuss the issues and recommendations." „ 



. Let 'us imagine that the Land' Owners, Organization Kas 
submitted a proposal and received funding for a small grant" 
from the Office' of Environmental Education to achieve the 
following : ^ - 

(1) To coordinate ^.community-wide gathering and. 
exchange of information related to land acquisition 
and usage along the Little Tennessee River . 

/ 

(2) To conduct educational activities ^hich ^nform 
• present and '^past landowners of legal rights and 

responsibilities in the on-going - River vs. 
Reservoir dispute. 



We have now created a situation similar to that of many 
OEE community 'education^ projects. ^ Within a small com- 
munity-action organization, a special project has been 
undertaken, a project primafily educational in intent, yet 
having a strong advocacy flavcfr^ The educational activities, 
or the "jjtfrriculum , are not theoretical, not academ^, not 
abstract? but are i*istead immediate and% tied closely to 
people's deep concerns and^ carings — in this cas6, for th^eir 
land. ' / ' ^ . ^ %^ 



Now let us consider evaluation possibilities for such a 
program. The evaluation situation is or\e which ^eems to 
encourage evaluators to shy away from measurement and to 
search for alternative, b^etter-fitting modes of data Col- 
lection and representation . They hope that such an eva- 
luation would: 

(a) allow for the study of open, changing 
systems and emergent problems;* 
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(b) encourage the representation v of multiple 
viewpoints and value perspectives; 

(c) focus on program activities and issues rather 
than outcomes; 

(d) provide a means of studying spontaneous 
events, situations, 'and crlse^ ' 

(e) be sensitive to/ the^cpntext and setting. 



Naturcdigtic Evaluation ♦ One approach whix:h seems to 
serve w^ll m rgeeting' -most of these hopes is naturalistic 
evaluation, sometimes called ethnographic or qualitative 
evaluation.^ Ethnography is a re"»earch technique adaptable 
to program evaluation from anthropological research metho- 
dology where it has long been applied in the study of 
cultural groups. Three des(driptors of ethnography are 
significant in^ understanding its utility in evaluation- It is:' 

(a) Field-based — Data collection is. conducted in which 
processes being evaluated occw. 

^ (b) Observational — A primary mdcie of data collection 
iT is direct observation of program processes ^ and 

evfents by tho' ev^luatoj or indirectly through 
interviews, dpcumecit an?ilysis, photpgraphy, etc. 

(c) Passive — In gathering data, the evaluator intrudes 
as little as possible ' upon the program activity. 
^ (For example, instead of administering tests, to 
* pi'ogram , participants or having them respcm^ to 

simulated problems , the researcher observes the 
behavior of participants as they*^ go about their 
usual program activities in usual program con- 
texts. ) ' . . , ' 



/ 





. . Case Study Research . One application of natunalistic 

evaluation techniques is in case study research. The -ease 
study approach to evaluation typically applies ehtnographic 

^ research techniques ' to the study of a single instance or 
case— or, for example, to the study of a small-scale pro- 
gram such as the Land Owners Organization. ^ 

» 

To understand how case "study research is conducted, 
let us first imagine shadowing an ethnographer through the 
various stages of a > case study of the Land Owners Or- 
ganizatipn. 

It, is' late in the fiscal year. Officers of the Land 
Owners Organization begin to plan toward a final report 
^ which soon must be presented to the funding agency of 
their comrhunit^ education grant. They decide to commission 
a case study evaluation of the L^d Owners Organization 
program and ^operations , especially, to focus upon its' com- 
munity education dimension. The evaluator, contracted 
frqm the nearly university, grumbles a bit at the late d^te 
and the organization's limited evaluation budget, but agrees 
to attempt a fifteen day, on-site case study of .the or- 
ganization and its educational efforts, as requested. 

\ Through telephone and mail exchanges prior to the site* 
visit, formal arrangements have been made * between the 
evaJuator atld the program staff which specify cost,- obliga- 
tion, access, issues 6f confidentiality, etc., for the case 
• study. 

J 

' ' The evaluator arrives on-site a day early, and utilizes 
the first aftern'oon and evening becoming oriented to the 
cultural and environmental context. She takes a leisurely 
di'ive along the Little Tennessee River, allows time to stop 
for viewing the valley, recently cleared for reservoir flood- 
ing. She locates a local newspaper to read, while having an 
evening meal at a local diner. Each of these experiences 
contributes to the development of a general feel for and 
understanding of the environmental milieu in which she will 
be working. ^ 

The first official contact of the study is with Al Davis, 
President of the Land Owners Organization. Excerpts from 
that interview follow: 

We have now 140. members . . .There are 683 heads of 
households (landowners along Little Tennessee) . Some 
didn't have to move. Only about 310 had to move. 
•T We have five elected officers. 



wSome people have been . hot^ for the [and for selftgh 
reasons. . , Only about ten percent are for it (thfe 
reservoir) • . . , * 

Waggoner (previous Director of TVA) wouldn't have 
even talked to you. . Now with Freeman (current 
Director of TVA) it's a (different ballgame . • . 

The Board of Directors meets every two weeks. We 
discuss problems — anything related to landowners • • 
recently, leasing, an^ the way leases are written, and 
about lowering the price of rent. We asked TVA to 
rqnt to former lai^ownerd first. We asked, for pro-- 
tection from poachers on bur leased lar>d- ^ Used ta^ 
you couldn't legally keep people off. 

We've met with Freeman twice. > In most cases they like 
us to meet with staff before the TVA Board - . • 
They changed the lease, but not suitably. 

Also, we •may be into 'other things, We want TVA to 
let the landowners have their land back. At 'this 
point, they can't let us . have it. As with thp 
Cross-6a'rge Cartal in Florida, the people have not 
been able to get their land back. A judge ruled that 
in three years if the project is incomplete, the land 
will be sold back at.* the original price. 



During this rather lengthy and intensive session, the 
researcher accomplishes several important tasks: 



She experiences a first exposure to the its history, its 
purposes, its activities, its problems and concerns. 

She identifies and gains information for contacting 
other informants: other Land Owners Organization officers' 
families who would -not move from the valley; sevet*al mem- 
bers who resist the reservbir plan; several members who 
ardently support tJtie reservoir plan; TVA officials; lawyers 
and other^ "experts'^ who have been brought in to give 
special presentations to the "^goioup and to local community 
audiences; key community figures, 

» 

She discovers the location and access routes to im- 
portant records and documents: The Land Owners Or- 
ganization Charter; " the proposal for the Land Owners 
Organization community education grant; ^maps; and docu- 
ments showing land divisions and ownership in the ^lley ; 
minutes f;:om the Land Owners Organization meetings; 
census information for tihe area, 
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She determine special events and activities which she 
will .want to observe: thfe Land Owners Organization ' 
monthly meeting; special education presentations* being 
offered during the fifteen-day period • w 



Our case study evaluator is still not qui)te ready to 
begin to gather data formally. At this point the evaluatoj- 
must determine some structure ^for organizing her data 
collection, Actqally, she usually prefers to use an 
issues'-based format, ^nd chooses to do so in her study of 
Land Owners . Organization as well, ^ Based on many 
factors—sensitivity to purpose and ' goals', awareness of* 
recent critical events and program process, current priority 
concerns of staff (as described by Al Davis), evaluation 
. audiences, and so onr^she arrives at the following list of 
issue-questions at least for starters; 

♦What effect have efforts of the Land Owners Organization 
had upon TV A, its policies, and its decisions? 

-♦Has- TV A exploite^^/itti^ organization for political purposes, 
or diverted the br||ni2:ation from its organization from its 
original committments ^d goals as an informal network? 

*How is the organization affected by community tendency to 
maintain the status quo? ^ ^ 

*Is the River vs Reservoir issue too sensitive to serve as 
the focus of an effective community education program? 

^. ^ ^ 
*Have organization efforts- been dominated by extremists on 
either side of the issues? 

*Is there ,an ethical conflict between the land-owners' deep 
desire to recover fheir land and OEE policy against political J 
advocacy 'as part of project activity? 
• / ^ ^ S 

She now \^ys out all of the possible data sources aJong 
with th* issues' to be investigated, weighs these against 
constraints, a:;id derives a feasible study pl^n. Once data 
collection begins, we see oUr evaluator coming and going at 
a feverish pace. vShe has set herself a rigorous schedule, 
perhaps overly full with interviews here and there over the 
entire county and as Jar away as Knoxville, filling the 
days, evening sessions and meetings, and still needing time 
to make data entries, to allow for spontaneous observations 
and experiences, and most importantly to maintain con- 
ceptual direction of the stijdy. 

The first phases of data collection seem chaotic, with 
the ethnographer experiencing-' considerable information 
'\ overload and disequilibrium. Slowly, however, the portrayal 
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begins to find its story line, to take recognizable shape and 
torn bpth in the data and in the mind of the evaluator . 
Perhaps the educational effort Within Land Owners Organi- 
zatioiff is merging beautifully with its political action thrust, 
with landowners becoming more involved and effective i^^ 
their voting knd action related to land use issues. On the 
other hand, perhaps the case study bears a story of mis- 
directed educational empowerment, with renters along the 
River becoming the real, but forgotten,- displaced persons; 
with Land Owner Organi:^ation membership consisting mostly 
of absentee urban landowners who participate in the organi- 
zation fgr political and economic reasons • And so on . . . 



So tKe ethnographer winnows the data, saving that 
which irfeems most salient, and weaving it into the fabric of 
a case study story. This process begins on-site, »and 
continues on beyond the data-collection period. 

The case study is synthesized into a final report which 
is then . xirciilated^^ tQ__appropria^^^ . Ideally , the 

ethnographer is able to return to the site to~revlew "ITie 
outcomes of the study with staff and possible membership of 
the Land Owners Organization. 



Perhaps our hypothetical evaluator had consulted and 
adapted (or her own situation a set of guidelines such as on 
the following page: 



I. 



II 



III 



IV ; 



V. 



VI 
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One'^et of Guidelines for Doing a Fie1d-0bi?ervation Case Study 



5fok« to -2,3 



ANTICIPATION 

Review or discover what is expected at the outset in the way of a case study. 
. Consider the ^questions, hypotheses, or issues already raised. 

Read some of the case study J iterature, both methodological and exemplary. 

Look for one or more studies possibly to use as a model. 

Identify the "case." Was it prescribed, selected to represent, or mere convenienc( 

Define the boundaries of the case (or cases),. as they appear in advance. 
Y Anticipate key problems, events, attributes , 'spaces , persons, vital signs. 
1 Consider possible audiences for preliminary and final reportings. 

F(jr-m initial plan of action, including definition of role of observer on site. 
Flk^T VISIT (?) 

Arrange preliminary access, negotiate plan of action, arrange regular access. 
Wrife d formal agreement indicating obligations for observer and .for host. 
, Refine access rules with people involved, including union, PTA, officials, etc. 
DISCUSS real or potential costs to hosts, including opportunity costs. 
Discuss arrangements for maintaining confidentiality of dbta, sources, reports., 
Discuss need for persons to review drafts to validate observations, descriptio/s . 
Discuss publicity to be given during and following the study. ' 
Identiry information and services^ if any, to be offered hosts. 
Revise plan of action, observer 's*^ole, case boundaries, issues, as needed 
FURTHER PREPARATION F0^« OBSERVATION " • 

Make preliminary observations of activities. Use other sites for try-outs? 
Allocate resources to alternative spaces, persons, methods, issues, phases/^ etjt. 

tdervM-f-y 4ftform<mts and sourc8S~af pArtiGuT^r -data. > .. ^ 

Select or develop instruments or sta'ndardized procedures, if any. 
Work out record-keeping system, files, tapes; coding system; protected 
Rework priorities for attributes, problems, events, audiences/, etc. 
FURTHER DEVELOPMENT OF CONCEPTUALIZATION \ - 

Reconsider issues or other theoretical structure to guide the data gati 



rr- 

/ 

rir/g. 



Learn what audience meftibers know, what they Wc^nt to dome to understand/' / 
Sketch plans for final report and dissemination of findings. ^ 7/ / 

Identify the possible"multiple realities," how p.>ople see things diff#en|ily. 




s ,etc. 



Allocate attention to different viewpoints, conceptualizations 
GATHER DATA, VALIDATE DATA ^ 

Make observations, interview, debrief informants, ga^ker logs, use s\ 
Keep records of inquiry arra'hgements and attivities. 
Select vignettes, special testimonies, illustrations. 
Classify raw data; begin interpretations. \ 
Redefine .issues-, case boundaries; renegotiate arrangements with hosts ;/as needed. 
Gather additional data, replicating or triangulating, to val idate /key /observations 
ANALYSIS OF DATA - 

Review raw data under various possible interpretations. 
Search^ for patterns of data, whether or not indicated by the issi/es 
Seek linkages between program arrangements, activities, and outcdfnes; 
Dray tentative concl usicins , organize according to issues, organise 
Review data, gather new data, deliberately seek disconfirmation fif 
PROVIDING AUDIENCE OPPORTUNITY FOR UNDERSTANDING 

Describe extensively the setting within which the activity occu|re( 
Consider the report as a story; look for ways in which the storft i 
Draft reports and produce materials for audience use. 
Try them out on representative members of audience groups. 
Help reader discern typicallity and relevance of situation as /Ibasi 
Describe methods of investigation and theoretical-^constructs ijfeedl 
Revise and disseminate reports and materials. Talk to people/ 



inal report, 
findings. 



"incomplete. 



for genral ization. 



erJc 



103 

r 

t 

There are always too many ways to spend case study 
research time. It should help delimit the study and allocate 
time to decide whether it is really the case or some pro- 
blems or issues occurring there and elsewhere that is the 
target of the study. If that particular cas^ is the target of 
interest, one should consider how to deal with issues vital 
to the case but of little interest to the inveiatigator, 
/ 

Those classrooms and other schools spaces are the 
living spaces of people. Most, by definition, are public, but 
private^ by common law. Observation studies can be an 
unwarrantedintrusion into the privacy of people. Seldom 
should the *rW^archer presume to be the guardian of ration- 
ality, efficijcncy, and morality, ^ 



The reader seeking understanding of the case or the 
issues ia usually not very interested in the methods of the 
study, pei-haps not sufficiently so. The researcher has an 
obligatiorv* to indicate the methods used and the choices 
made, bM|t these statements should not be so placed as to 
obstrucb^r.the reader's effort to understand the substantive 
findings^! 



Thp reader who wants to challenge the findings is 
likely t*ij sAy that the case study was subjective, arbitrary, 
non-rejjwesentative, and inconclusive • Which is probably 
true,'>^t the study is not thus invalidated, .The counter 
to thedwl charges, if strong efforts to produce a valid, study 
have oc?curred, is a good description of the methodological 
and conceptual reasoning that took place, including efforts 
at verification and .disconfirmation • 



le j feselyrcher should reflect Vpon his/her own com- 
mitfhents Und biases, No observation x^r case study is free 
of bias, nor is that a critical aspiratioiv^^^%<il the validity of 
the study is usually increased if the reader>san appreciate 
the bias, ^Two ways to do that are to descrime nationwide 
problems as they valued personally by the researcher and 
^'^TO describe in the reports some local situations that are 
relatively common — ^so the reader can see how the record is 
fnfluer^ced by the researcher's commitments tLXid biases. 
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The Project Historian . The project historian is a 
variation of case study e valuator. Such a person can play 
a special, evaluation role. She may Vitilize ^ethnographic 
•research procedu,res> incorpiprating them into the on--.going 
process evaluation of a program. 

• i 

The project historian observes, interviews, or searches 
the documents, and generally immerses herself into the 
stream of events of a program. i4owever, the evaluation 
process extends the full duration of the program protess, 
and interacts with it intermittently. Ifi addition to eval- 
uating the progrram through ethnographic inquiry methods, 
the project historian has the added responsibility of 'pro- 
viding on-going feedback so that the eValuation results can ) 
have immediate utility in prograjjn planning, 
decision-making, and direction. 

Th« Land Owners Organization might choose to engage 
"a project historian in the evaluation of their progrjun. 
Usually the evaluator would be a volunteer from the com- 
munity, a retired member perhaps dr a collegye faculty 
member. Such a evaluation would entail intermittent visits 
to the site for observation, interviews, and review of - 
program documentation. The evaluator might require time 
^t Board Meetings for the exchange of information pn Jthe 
progress,, problems, etc. ^ of the project. * • ^ 



Cautions about Naturalistic Evaluation . At least three 
special problems are associated with naturalistic evaluation : 

(1) These studies are expensive. They often i^ivolve 
evaluators for seemingly inordinate durations and at 
great time and financial expense to programs. 

(2) Such studies require ' evaluators witii , speciati* 
orientation toward and preference for naturalistic (as 
opposed to' experiinental) studies, as well as some 
experience in field-based research. Such evaluators 
are in a minority across the field of evaluation, and 
can tie difficult for a program to locate. 

• (3) Results of naturalistic studies are inappropriate 
for many audiences and evaluation purposes . They 
often appear * overly subjective, arbitrary, arid non- 
representative . ^ They arfe unresponsive to audiences 
with a penchant for experimentally-derived data. 
Results of naturalistic studies are often slow in com - * 
ing , and once in hand , of great, length, detail, and 
complexity. For this reason, they fall to meet eval- 
^uation needs which demand immediate, concise, and 
.conclusive results. 
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ORGANIZING AN eAlUATION STUDY 



We have used half the- space of this Program /Evaluation 
Manual telling you about LTVEC and the ^tizens League 
because we feel that evaluation studies should be rooted in 
their programs, sensitive to the local context, and 
structured upon . the issues* thajt concern people near at 
hand* It is^ natural for persons charged with difficult 
evaluation responsibilities to look for research designs and 
evaluation models tested, validated, and made ^ready for 
general use. Such designs and models exist. Th'ey have 
been proposed by many specialists. But actual usage has 
shown them lacking, needing adaptation to local conditions 
and extension to otherwise overlooked fundamentals of the 
program. It can be shown that the proposers of these 
designs, when ciarrving out evaluation studies themselves, 
do not stick closelyl Ifo their prototypic designs but draw 
eclectic ally^ from a nu)nber of different ideals. 

What the successful evaluation specialist appears to do 
is to think simultaneously of the* uniqueness' of this program 
and potential modification of designs that work best for him 



attending to bot^ the ^real and . the ideal, drawing other 
people into the planning, the experienced evaluation arrives 
at a creatively tailored /plan , T^at probably is what in- 
experienced people should do too. 

Before we get more into the detail of organizing an 
evaluation study it should be useful to consider some of the 
major concepts , ajid prototypes to be found in prggram 
evaluation work , 



or her. 




forth in some kind of dialectic. 




\ BASIC IDEAS* OF EVAJ.UATI0N QF EDUCATIONAL PRO- 
GRAMS ^ ' 

v. ' ^ / J 

A program, like a hurjian being, is a complex creature. 
There are so many things about it that could be studied. 
Often the expectation ia that evaluation attention will be 
'divided^-up among lots of features, but that seldom works. 
To get a successful evaluation study you usually have to 
focup attention on just a few things. You have to delimit 
the/ stu.dy. One common way to delimit the stuc;iy is to 
ide whether to focus on the workings , the process of 
A. program, , or its output, its impact. Those studies that 

aim to examine the relationship between process and produc^t 
/ do so hy only considering a very limited view, of .each , and 
then usually giving attentix^n to what seems to explain a 
single dimension of gutput. 

Process versus Impact . It seems so obvious that any 
good evaluation study would want to examine both the 
process and impact of a program. Well designed studies do 
attend to both, but almost always one dominates the o^her. 
It may be that the evaluation activities for studying process 
and impact are so dissimilar — and each so demanding of 
resources — ^^that evaluation designs almost always emphasize 
one more than the other. / 



In the first chapter we presented a letter from a 
regional program official expressing a need for impact data. 
As the distant administrator thinks about the program a 
simple decision model often is apparent—if impact is high 
enough and costs are low enough, (decisions should be made 
to sdstaiq^ or expand the program. Those decisions do not 
require much knowledge of program operation, so the re- 
gional official may stress the need for impact over process 
knowledge . ^ 



Of course it is not only distant ^administrators who say 
that evaluatioji should focus on^^program impact. Tax- 
payers, developers of new program^, supporters and critics 
alike are quick t(v ask for evidence of accomplishment. It is 
common in Western technological countries to think that a 
collective effort, a public enterprise, should produce some- 
thing. So evaluators are urged to look for impact. 



But somehow there is a general Inability of educational 
researchers to satisfy this natural and apparently reason- 
able demand. Why don't all program evaluation designs 
concentrate on impact? There are several reasons: 
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(1) For all the technical sophistitation of eduational 
testing the results from the existing standardized 

^ tests are usually too crude and not sufficiently on 

target* Real attitude ^change, understandings, 
political conviction, and group effectiveness-*-all 
important program impacts^-are different to 
measure meaningfully. And to develop moi*e 
refined tests is a very expensive task. 

(2) Standards of success ^ differ from one place to 
another so that what wiUxbe considered successfful 
impact here will not be considered successful 
elsewhere. ' • 

(3) Many effects are ^ expected to develop slowly, 
providing at the time of evaluation very little in- 
dication of what will come .later. And when that 
later measurement comes there may be a number 
of causes to which to attribute the results. 

N ^ . 

(4) The cause and effect relationships, betWen 
process and impact are Joo complex, too sltua- 
tiojial. Even when we know what the effects are 
we may have little idea about what to do about it. 
Data on effects just do not reveal much about 
what caused success or failure, and even less 
about what changes would increase the sCccess 
and diminish the failure. 

So the regional official who wrote the letter may succeed in 
getting that evaluation design more oriented to impact, but 
may be disappointed with the degree to (vhich impact is 
actually measured and with what can be done once impact is 
measured. 

f 

We should acknowledge that the letter was a response 
to evaluation design which was heavily oriented to issues 
^nd processes. Its author might have been just as quick to 
seek correction for a design exclusively attentive to impact. 



In a few pages we will present a summary of the majo^ 
approaches to program evaluation. The primary basis we 
use for classification is the distinction be'tween 
process-oriented and impact-oriented designs. 

Preordinate versus Responsive Designing . The dis- 
tincti«n between "process and impact is made as to what the 
evaluation study is looking at. The primary distinction as to 
how the evaluation is carried out perhaps is that between a 
preordinate approach and a responsive approach. 



1Q8 



Studies differ as to how much the inquiry is based on 
preconceived questions or hypotheses and >iow^ much on 
issues encountered in the field during the study. These 
are sometimes referred to as etic issues and emic issues , 
respectively. ' 



The responsive study is develqped\ slowly, »using 
Malcolm Parlett and David Hamilton's^ conc^^t of pro- 
gressive focussing , basing fui«ther inquiry on observation 
of activities and the emergent concerrjs of .participants m 
those activities. 



Preordinate studies are more oriented to objectives, 
hypotheses and prior expectations, usually expressed in 
formal statements, Preordinate evaluators know what they ^ 
are looking for and design the study to find it. 



In a oreordinate study^ a relatively large portion of 
resources is spent getting objectives specified and de- 
veloping instruments, and sometimes in^ettifig up control 
groups and randomly sampling* In a responsive study a 
relatively large part of the*||^source pool i^^spent preparing 
and placing observers and interviewers in the field, 

N 

Wholistic versus Analytic ^ Perception^ , Studies differ 
also as to how much they treat the program as a totality, 
recognizing conceptual bounc^ries common to non-technical 
audiences. The more common social-science research ap-- 
proach is to conce^ntrate on a small niumber of components 
or prope^jties, A case study is often used to preserve the 
complexity of the program as a whole. Analysis (often 
called multivariate analysis ) is undertake^n to study the f 
relationships amohg properties. 



Formative versus Summative Commitments , The most 
enduring distinction made by Michael Scriven in his respec- 
ted paper "The Methodology of Evaluation" is that between 
the formative and the summative. He was particularly 
distressed by the over-attention evaluative researchers 
seemed to be giving to" the concerns of developers of pro-- 
grams (formative concerns) and the lack of attention to the 
concerns of consumers such as students, parents and other 
citizens (summative concerns), / 

/ 

The temporal distinction between evaluation during 
program development and that after completion is usually 
emphasized, but the question of whose concerns will be 

Ho * 



109 



addressed is probably more important in considering 
whether to organize a f<;jrmative or summative evaluation 
study. 

Formal versus Informal Inquiry . This distinction was 
made earlier. Informal evaluation ik a universal and abkling 
human act , scarcely separable from thinking, and feeling . 
Formal evaluation is blue^printed, operationally de- 
fined — less intuitional, covert, and personal. ^ It is needed 
when the results are to be communicated elsewhere. 

" yConsensus versus Pluralistic Value Sets . Gene Glass* 
*and Fred Ellett based a recent taxonomy of evaluation 
designs mainly on the distinction between those which tried 
to find the coramon valuing across ftmes, places, and 
people, and those which tried to. preserve the differences in 
valuing. The important emphasis here isT more on the 
criteria (What is the definition of impact and cost?) more 
than on standards (At what \ev/kl shall we say that the 
program is sujccessful?) 

Case Particular Knowledge verstu| Generalizations . 
Perhaps the thing that most differentiates between what we 
might call evaluative research aJid program evaluation is the 
ultimate purpose of inquiry. ^ The distinction here is similar 
to that made by Lee Cronbach and Pat Suppes in dis- 
tinguishing between decision-oriented inquiry (for the 
practitioner mostly) and conclusion-oriented inquiry (for the 
^.theorist mostly). 



Some studies are set up to examine the program as a 
fixed and ultimate target, others ta^. examine the program as 
a representative of other programs. Most research is 
expected to generalize in somjs ways: over time, over set- 
tings , or over subject matters , for example T he inquiry 
may aim at the worth of the particular program or at the 
worth of the general approach. Studies are conceived very 
differently in this regard, both by investigators and by 
audiences — a large misunderstanding is possible. 

. The more the study is expected to be a basis for 
•generalization the more the need for controls, controlled 
variation, and careful description of uncontrolled variation. 
Description is needed of the changes in time and place and 
persons, and in many of the ways in which generalization is 
to be directed. Case studies may be undertaken either for 
particularized knowledge or for generalization. 
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Descriptive versus Judgmental Interpretations . As 
indicated in the first chapter, all evaluation is judgmental . 
Studies differ however on the degree to which the research 
includes and even emphasizes his/her own judgment and the 
•degree to which readers are left t6 (and assisted in) 
making their own judgments. 

Many evaluation specialists coming from a social science 
background define the evaluation task largely as one of 
providing information, with an. emphasis on objective data 
and de-emphasis on subjective data. Those with humanities 
background tend to reverse the emphasis. 

Phere^re really two separate questions raised here , 
whether or not subjective data (testimonies, preferences, 
etc.) will be used, and whether or not the judgments 
formed by the evaluation team will be reported. 

I 

With this dichotomy, and each of the others above, we 
have a continuum. Mixtures are not only possible , they are 
the rule. Studies at either extreme are seldom found. The 
extremes are cited ^ here to identify dimensions or 
characteristics by which a new study can be conceptualized. 
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A CLASSIFICATION ' OF PROGRAM EVALUATION AP- 
PROACHES . • 



. ^, 



Certain ways of carrying out evaluatiort* studies have 
become commonplace. The table on the next -page presents 
four major categories of evaluation studies and some special 
cases within eac^. These prototypes should^ aid in the 
discussion of the altertwttives available to 'program staff 
considering possible ways tg^ carry out' an^ evaluation study. 



In the previous chapter ,we examined five evaluation 
situations. Two of them were * directed toward program 
impact ^ one of those to the attitudes of teachers in a con-- 
tinuing education situation, the other to knowledge attain- 
ment of students. What is common to these and to' most 
impact study situations is ^ that people^ identify one or a 
small number of '^outcome variables" and organize data 
gathering around them. OtKer examples of program outcome 
variables are : attendance at meetings p referendum votes > 
newspaper^ ^tjoverage^ familiarity with the environmental 
problem, and donatioi;is. 



We take note of t^A(o kinds of impact designs: 

a. Gains in performance over time 

b. Regression of background and process 
variables on outcomes 



The approach usually suggested by measurement specialists 
and educational psychologists is "testing to measure student 
gain . " It jrelies on instruments developed or selected to 
match specifications of objectives. In the special case of 
the Goal- Free apj^roach the specification of objectives may 
be informal and made by * the evaluators rather than the 
program staff. This study is likely to give the most ac- 
curate indication of student (or other beneficiary) impact > 
but least likely to indicate what caused it or how to improve 
upon it. 

Those studies designed primarily to explain the impact 
of the program are Regression Analysis studies. The term 
is a generic term, not limited to those where - a formal 
statistical test of regression effects is made. Any effort to 
study the covariation or contingency of process with out-- 
come can be classified here. The strength of the approach 
lies in its accuracy and control, the weakness in its assump- 
tion that the things worth attenti6n will show 'up having a 
variation of scores available for analysis. 
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A CLASSIFICATION OF PROGRAM EVALUATWN APPROACHES 

/ ' Disadvan- 

A leading advocate/An exemplary study/Advantaqes & tages 



PFRFORry\NCE GAINS ANACYSIS 
• Special 



AIM: Assess im- 
part of program 
with regard to 
prespecified 
criteria 

RFGRESSION ANALYSIS 

AIM: Find signi- 
ficant covariates 
(background or 
process) of out- 
come variables 



Cases 



Spe(;ial 
Cases : 



Discrepancy Evaluation 
Field Experinlehts .... 
Goal -Free Evaluation . 

Survey Sampling ...... 

Cost-Benefi t Study . . . 
Explanatory Observation 



Ralph Tyle/" ., . 

Malcolm Provus Torsten [#usen 

Bobert Boruch^... Dick Anderson 

Michael ^criven ... House/Hogben 

Lee Cronbach ... 

Hubert Blalock Daniel Weiler 

Henry Levin Fielden/Pearson 

Bill Cooley ....... Cooley/Lein- 

hardt 



MANAGEMENJ INFORMATION i 

, ' Special 
AIM: To keep good Cases; 
records , increase 
, rational i ty in 
decisions * , 
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NATURALISTIC STUDY 

AIM: Provide ex- 
perleiltial under- 
standings of com- 
plex workings of 
the program 



Special 
Cases : 



... Peter Rpfr^i ... 

Policy Analysis James Coleman Haveman-Watts 

* Dfcision-Theoretic Marcia' Guttentag .. Edwards, et.al. 

Adversary Evaluation Tom Owens Murray Levi ne 

, . . .( Fgon Guba ... 
Self-Study J^aul Dressel Knoll/Brown 

Blue Ribbon Panel Francis Chase Plowden Report 

'Ethnographic Case Study . Lou Smith Stake/Eas^ley 



Keeps attention on the 
"bx)ttom line," invpsts 
in performance testing; 
Oversimplifies educa- 
tional aims , ignores 
ways to improve^rocess 



Some control of condi- 
tions by researcher 
^^^using robust sampling 
plans, analyses ; De- 
emphasis on educational 
issu«, particularly 
local circumstances 



> 



O 

00 
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Emphasizes policy, 
quality-control options 
of administrators, boards; 
Over-values efficiency, 
undervalues tacitrknow- 
^ledge of staff, constiu- 
ents . 



UJ 

o 
o 
or 

Ou 



Probes complex concerns, ^ 
responsive to emic issues;^ 
Subjective, poor meta- 
evaluation checks, unat- ^ 
tentivQ to, aggregate costs J 
and benefits r^, 
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Similarly, we take note of hvoykinds ot process designs: 

a. the representation of- program ^'information in 
testimony and indicators for management 
systems 

V 

the naturalistic and experiential study of 
ordinary program activity 

^ management information system is attractive to many large 
organizations. It is expected that the worth of various 
organizational efforts can be better judged if there is a 
regaUr flow or special study of data related to the opera- 
tional and policy decisions managers are faced with. It is 
generally expected that personal experience will fall short 
of the knowledge and understanding needed and that formalis- 
tic and highly conceptual (probably abstract) r'epresen- 
tations of the program are needed. • 

Naturalistic studies are different in that they con- 
centrate on the more ordinary perceptions of practitioners 
and citizens. Skills of ^Wservation and interviewing are 
usually needed. The discipline of this inquiry comes in the 
validation and interpretation of data. Self-studies are 
among the least expensive but ethnographic case studies are 
among *ie most expensive. 
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GETTING ORGANIZED 



In most Environmental educatio/ proiects there i., no 
evaiuators in this manual, knowing that what is to he 
tterv^::i.h T^'"^ ---^-^tion study , often d^nf en! 

tire y without anyone with special training. Consultant he^D 
or technical assistance may be available .but even then the 
toughes part of the evaluation work will probably be done 
by people already in and around the program 



0 



^he tir's't chao't'er ^""^ J^S " ^'^'^^ 

* needL- 3 someone decides that an evaluation study 

• needed-internal or external pressures call for it A 

Tt^I bloT"^ "''^^ ^''^^^^ 8— ^ knowledge^'" 

full blown evaluation studies require highly trained re- 

TrZtrdo"^ ' °' extra-large 

projects do not have a large enough budget to carry out a 

r To'r tLTn 7*^r°" ' 'ittle^one;:Te'rtX; 

ab e AnH^ °( operating costs might be made avail- 

able. And some volunteer labor, some" from staff people 
fron. students needing a graduate school project, from ot 

resouTce"Tnt™"^'\"'"^"^' ^« ^ohnLd. A sZl 

resource, not enough to satisfy critics or to authentirafe 
proponents-but enough to do something worthwhUe ' 



the ^ °f- questi^s is to decide whether to keep 

vthe evaluation responsibilit^inside or to farm it out to 
somebody outside the refdar program ^^oup To 

Hr^J^lf fit th r r^'^^ tLtT;eds to 

De made tO fit the local situation. 



r^r-i ^^«°if^tina; a Contrnrf. if outsiders are to take a 
primary-responsibility, for- the evaluation, a writ en a«ree- 
men is highly desirable. In addition to the usui sUte- 
ments of purpose delivery dates, personnel, and budget 
the parties should anticipate that things will go badiv and 
mdic^^te how adjustments can be made. "What co^d 
.wrong will go wrong- is a .useful slogan. ^ 

A contract can be too specific and not specific enough 
Wien the contract is too specific the eVa'luator will 
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unable to respond to unexpected difficulties — and op- 
portunities. If the contract is not specific enough the 
chances of sponsors getting what they want are reduced. 



The written statement should be based on many de- 
tailed qonversations , providing expectations that go beyond 
legal obligation. ^ But conversations that get too detailed are 
likely *to de-emphasize the larger policy questions, to disr 
dain the more subtle "unmeasurabl^" circumstances, and th^ 
parties are likely to over-protect themselves with con- 
straints and risk-lowering, clauses that could limit the 
usefulness of the study. If, as usual, the conversations 
are not specific enough, there is reduced^ chance that the 
study will engage some of the deeper concerns of people* 

Perhaps the most important question to be raised, once 
and again and again, is that of the purposes of the eval- 
uation. Of course, the purpose of evaluation is to find out 
what is good and what is bad. But Irtie particular questions 
of concern to participants, , sponsors, and. other con- 
stituencies need more than final, judgments. Tchey need 
information, interpretations, suggested alternatives. Which 
of these purposes shall be emphasized? 

X 

Another question to be agreed upon is that of who the 
audiences of the study will be. To whom will the results 
be cifcuTated? Often it is difficult to identify an audience. 
O^ten it can be expected that no-one will read the evalua- 
tion report. Yet several groups may be influenced by the 
findings ♦ Both attentive and almost unknowing sudiences 
need to be discussed* 

Methods of inquiry, access to data sources, ^on- 
fidentiiality , rights to publish, and ways of amending the 
agreement are other topics of imp9J*tance* 
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A PLAN 'OF ACTION 



It is usually important to anticipate the i*esponsibilities 
of the , evaluation study by sketching out the steps to be . 
taken. In hi3 discussions of responsive evaluation. Stake > 
used a clock as a metaphor and identified the twelve events 
shown in the following figure. ^ " 

C' 

It was not expected that an evaluator heeding this 
clock would take the steps in strick order but going back 
and forth frequently between earlier and later ,^ps. 
"Progressive focussing" Malcolm Parlett and David Hamilton 
called it. Nor was it expected that equal time would be 
spent on each. One study might devote 80% of its time to 
making observing program activ^ities, whereas another 
devotes less than 20%. 



A number of the same topics appropriate for the nego-* 
/tiated agreement (as described in the previous section) are 
appropriate for the plan of action. 



It is wise to have a number of people take "a look at 
• the plan of action and make suggestions* for improvement. 
Sometimes ^''it is useful to give them guidelines for their 
critique, such as the checklist that follows the "responsive 
clock . " f 

The evaluator should -^eep a log (or at least a cal- 
endar) of important occasions, particularly points at which 
decisions were made. At the conclusion of a stud-}- the / 
evaluator often has a different recollection (or no recol- 
lection at all) of matters that influenced the study greatly. 
Lines of inquiry that emerged or were dropped in midstream 
should be described in the log or newsletter or some such. 
Thefse records are valuable often to the evaluation audiences 
as well as to the evaluation staff. 



) 
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Format for 
audience use 



\ ' Talk 
with clients, 
program staff, 
audiences 



Identify 
program 
scope 



" Winnow, 
match'- issues 
to audiences 



Overview, 

■k 

program 
activities 



Validate; 

confirm 
attempt to 
disconfirm 



Discover 
purposes, 
concerns ' 



Thematize; 
prepare protrayals, 
case studies 



Conceptualize 
issues, 
problems 



Observe 
designated 
antecedents 
transactions 
and outcomes 



Select 
observers, 
^ judges; 
« initruments 
if any 



Identify 
ds/La needs, 
^n. issues 



Figure 2. Prominent events in a responsive evaluation. 
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^UlOE FOR RATING AN EVALUATION PLAN 

4t 





Strong 


Satis- 
factory 


-1--^- 

Weak 


COMMUNICATION- , ^ , * 








Clarity: Does the plan read well? explicit? "free of jargon? ...... 

Integrity: Do its pieces fit together? sense of uhity? . . . : 








Seductivity: Does it persuade- the reader to take part? intriguing? 








PURPOSt ' * . 








Rationale: Does it answer "Why evaluate?" are goals explicit? .... 








Plan: Are atti vi ties outl ined? ard plan and rationale consistent? . . . 








^ Complementarity: Is it compatible with program? with other studies? * 

« 








TECHNIQUE 








Variable'selection: Are selections relevant, iioaginati ve , flexible? 








Data gathering: Are efforts design-oriented, valid, redundant? .... 








Reporting out: Are clients, audiences specified? responsibility? . . 








PRACTICALITY 








Scope: Is the plan feasible? are staff, facilities available? .... 








Lost. Are estimates reasonable? are benefits worth cost? . 






• 


* i» 

Balance: Are resources nicely allocated? spread too thin? 








« 

COMMENTS: " 
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SUBSTANTIVE QUESTIONS 

Certainly for a preordinate evaluation study and 
usually with a responsive study the evaluator wants to 
identify in advance certain topics, issues, or substantive 
questions. It usually is necessary to have routine re- 
minders of key Nsubstantive concerns because the problems 
of arranging the\ study tend to distract attention toward 
methods and away from those concerns. 



These organizing questions may or may not change 
during the course of the study, depending on the design. 
The questions will vary in specificity depending on bow 
intrinsic the evaluator' s interest is in the particular pro- 
gram and in tei:;ms of how well the program is known in 
advance. 



The following is a simple classification of organizing 
questions, with illustrative questions: ^ , 

Simple description 

, Describe the program 
How would one classify this project? 
How much public support (or any other 
property) does this program have? 

Comparisor^ description 

How does this ^ program compare to others? 
Is there much variety among the projects? 
Has this program changed over time? 

Description of objectives 

What is this program supposed to accomplish? 
How do people differ in what they want it to 
^ do? Is it feared that, jobs will be lost? 

Descriptions of judgment 

Did this program do its job well? Was the. 
publicity given its work adequate? 
Who supports and., who opposes further 
environmental education *of this type? 

Inquiry into issues 

Should these teaching materials be em- 
phasized as a s^l f-contained course or 
as adjunct lessons to science and social 
studies courses? 



2. 



3. 
4. 



A CHECK LIST FOR RATING AN EVALUATION REPORT 

f 



Needs * 
Well Better . Not 

Stated Statement ^ Stated 



^ Not 
Appli- 
cable 



Area I-THE EVALUATION ITSELF ^ . 
-A. Audiences to be 'served 6y the 
evaluation 

B. Decisions about the prograiTI, 
anticipated 

C. Rationale, constraints, bias ol* 



,,>e valuators 



B 



Area II--SPECIFICATIONS OF THE PROGRAM 
BEING EVALUATED 
A. Educational philosophy behind 

the program 

Subject matter to be taught . . 

C. Learning objectives, staff aims 

D. Instructional procedures, 
tactics, media 

E. Students: biography, readiness, 
goals, etc 

F. Instructional and community" 
setting 

G' Standards, bases for judging 
quality 



Area III — PROGRAM OUTCOMES 

A. Opportunities, experiences 
provided 

B. Student gains and losses . . . . 

C. Side effects and unexpected 
bonuses 

D. Costs: cash, resources, work, ■ 
morale 



Area IV--RELATIONSHIPS AND INDICATORS 

A. Congruence between intent and 
actuality . , 

B. Contingencies, causes and 
effects 

C. Trend lines, indicators, " 
comparisons 



Area V--JUDGMENTS OF WORTH OF THE 
PROGRAM 

A. Value of outcomes, different 
points of view 

B. Relevance of objectives to 
needs 



of 



Readabi 



of report 
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Usefulness of evaluation Information gathered 
Conments : -f o 
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Are good - steps being taken to keep study- 
group discussions free from domination 
by special interest groups? 
Is it time for the group to rethink the unde- ' 
sirab^ty of burning high sulphur coal 
^ . as an alternative to nuclear power? 

These latter substantive issues may >be particularly 
useful for organizing the evaluation ^study, if they dp in, 
fact draw ^attention to what concerns people^. Sometimes it 
is also desirable tht they draw attention to Vhat feature or 
component of the program might be adjusted to improve the 
results • A X 



The development of a list of substantive questions 
helps the evaluator inform people what the evaluation is to 
attend to. But more important, it helps the evaluator get 
back on the tract when distracted. It is expected that the 
list will need updating from time to time in many studies, 
whereas others wilL be designed to stick with the questions 
chosen in advance. 



RECORDS AND REPORTS 



The evaluator is in a perpetual dilemma between an 
overlpad of data and an inability to substantiate con- 
clusions. Fancy coding and retrieval systems are sometimes 
devised, but- their expense (mostly in time) is seldom just - 
ified. File folders and^hoe boxqs ar^j|foften as sophisti- 
cated as ^he data bank should get, storing things chrono- 
logically and by site.. By frequently trying to identify 
emerging conclusions, and asking what additional data are 
needed to round out and * validate those conclusions, the 
problems of substantiation (vaiidati on) can be reduced. 
Norman Denzin Jhas described the use of triangulation for 
substantiating one's conclusions. 

.\ 

In some studies "W^ich story to tell?" becomes a very 
difficult question. There 'may be "multiple realities" that 
nei&d to be realized. Ther*^ are bound to be some issues 
that have to be omitted because they are too distracting or 
for which the observations are inadequate. The evialuator. 
decides on what to report partly by trying oyt-^lie alterna- 
tives (In various ways) on individ ual c w tfo represent au- 
dience * croups. This is not to suggest that we tell them 
what th^y want 'to hear, but we try to honor their prior- 
ities of concern. 

i3o 
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ORGANIZING A RELS EVALU^ATION UNIT"^ ~ 

Given the complexity and interdependence o4 con- 
temporary environmental issues, it is apparent that we face 
a need for new forms of sociil org)i^nization. A noted 
scholar in the realm of social organization,^ the late James 
Thompson, anticipated such demj^ids in an essay > entitled 
ORGANIZING SOCI£TY»S FRONTIERS, Thompson Stressed 
the importance of "slack" in the system, that . is, main- 
taining an organization flexible enough to react in creative 
fashion to «new problems. He warned that demands for 
accountability, "getting rid of slack," would increase, with 
reliance on new abstractiona of the problems (social in- 
dicators) increasing too. 




Changes in policy at the U • S, 
Office of Environmental Education 
indicate that projects funded in 
the 1980s will probably be larger, 
more comprehensive, and sup- 
ported for more than a single 
year. The agenda outlined below 
assumes that RELS-like projects of 
the fiiturp are likely to spend 
several thousand dollars per year 
for evaluative^ services. V/hat can 
project directors challenged to 
produce evidence of maay kinds 
expect to receive from evaluators? 
How can both parties honor 
Thompson*s recommendations? 



Getting program evaluation is 
not unlike getting informa^on on 
regional • environmental problems. 
Much of the infcrmation needed is 
known by someon||t in the region. 
■Most of the valae positions are 
represented thate in the com- 
munity. The problems of program 
management, can be addressed%jDy 
collective inquiry, and can be 
served by the technology of in- 
quiry* 
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Thus r in " a senge , an ev^altia:tion--tmit is^^ a mini-RELS . 
It should be a flexible system of inquiry by which one can 
come to better understand the workings of the organization 
in its community. The inquiry technology of the RELS 
should be a major asset for program evaluation. 
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EVALUATION RESPONSIBILITIES 



Whether or not a formal- evaluation , unit is set up, 
people in and around the RELS will see a need for formal 
evaluation of some activities. It may be useful to consider 
what some evaluation respcm^ib'ilities are likely to be. 

Needs Assessment . In any organization and in any 
community there is a reaching for clarification as to what is 
really needed, and what are the priorities on those needs. 
It has become routine for spokespersons to announce that a 
formal study of needs, is essential. It seldom is qssential, 
and it may do more confusing than clarifying, but a needs 
assessment study sometimes is desirable. 



In our experience, large amounts of money haj^Cbeen 
wasted on poorly conceived needs assessment studies. 
Assumptions are made that what is neede^l under present 
conditions would also be needed under other conditions. 
Most needs are really ■ wants , still terribly important, but 
not as. critical for subsequent workings as the name "needs" 
Suggests. Need assessments are too often derived from goal 
statements, which though stated thoughtfully and sincerely, 
draw attention to arbitrary and unrealistic ideals and away 
from the responsibility for alleviating problems. (See Stake, 
asley and others; their page 19:18.) 



Resource Inventory . < °The most attractive treatment for 
^ problem' is often beyond , the resource capacity of a local 
community. Before we tackle a problegi it is important to 
be realistic about what resources cart be mobilized. Part of 
the evaluation responsibility is some form of feasibility 
study, possibly quite informal, noting in part whether or 
not resources ar» available to undertake various efforts at 
alleviation. 



This is not meant to be an advocacy for^those remedies 
for which we are best 'equipped (as the specialist who 
always finds his speciality most appropriate, or as the 
drunk who searches Under the streetlamp where the light is 
better). What remedy to try depends of coui^e on what 
tools and talent^are available, but not necessarily, most 
available. . ' 

Goal Revie w.. In common with social endeavors, RELS 
are likely to be ondq^ed with a 'vast multiplicity of goals. 
Choices must be made, trade-offs must occur. Formal 
evaluation can assjist with the delineation of the goals of the 
RELS, just as the RELS assists in the delineation of 

1 '> 
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community goals. One evaluation approach taking a de- 
liberate orientation to the suitability and relative merit of 
igoals is the CIVP approach put forth^by ligj^ Stufflebeam 
and his colleagues. 



A goaJ^ review that attends only to the ultimate aims of 
people is lively to be of little value. A thorough study of 
goals examines the nature of major problems and the nature 
of proposed treatments. The victims of each problem have 
goals needing examination and the advocates of each 
solution have their goals too. ^ 

Program Transactions . The RELS needs to be studied 
continuously by its staff, participants and patrons, and in 
special ways 'on special occasions. Processes can be need- 
lessly inefficient and ineffective, but efficient and effective 
processes ean be largely irrelevant. 

The aims of environmental education programs often 
embrace individual and institutional goals. Both may re- 
quire baseline information for assessment of success, * ' ' 

A document released in 1976^ by the U.S. Department , 
of Transportation declared, that "asTcing appropriate ques^ 
tions" is the key to assessment- of programs marked by^ 
broad-aim goals. We second that advice, noting that it is 
not the elegance nor the' precision of the qu^tiort that 
counts, byt it^ relevance. Since early and relatively minor 
transactions often have a lasting influence, much persistent 
questioning aout transactions can often be justifiecj. 



Predicaments . The RELS will get into trouble.* Part 
the responsibility of the evaluation unity will be to studjf 
rouble and assist in the extrication. 




It is obvious that the staff and directors pf the RELS 
will fce attending to predicaments, as well as to the other 
fouryeategories df responsibilitfy detailed above. Evaluation 
responsibility is shared by all, and most of all, by manage- 
ment people. The evaluation "unit" should be. seen by them 
as collaborators in the study of the »organizatiph. 




But the evaluation "unit" is a spec^l collaborator. 
Certainly it has constituents, audiences, patrons other than 
the RELS managers, ' It must review value positions that 
are unpopular among managers , as well as favorites. It 
must seek to tell managers what they need to know, cer- 
tainly not only what they want ,to hear. 



THE STRUCTURE NEEDED ^ 

Most RELS will grow out of some other organizations 
and activities. Some of those will have a designated evalua- 
tion responsibiUty already. Most wiU not. And most RELS 
will get started without a designated evaluation person or 
unit. Which is all right. 



There will be 'teome RELS that start with a grand 
design— particularly those seeking funding from the U.S. 
Office of Environmental Education for comprehensive 
multi-year projects. And as part of that grand design 
propose a builVinr evaluation unit.« Which is all right too. 

Reasons Against an Evaluation "Unit" . It will be easy 
plan big, to promise too much. There will be a temp- 
tation to include a design for an Evaluation Unit that will 
cost more than it will be worth. Evaluation information of 
high reliability and validity is expensive. And an evalua- 
tion unit that is set up before knowing what the infarmation 
needs and pjroblem situations will be is likely to have the 
wrong kinds of assets. , 



Whether or not to establish a formal EVALUATIOl^ 
UNIT then depends partly on how much there will be a 
routine need for evaluative information, that is, for attitude 
scores, fo,r monitoring and quality control, for project 
history, etc., on which judgments can be based and for 
reports to funding agencies. To the extent that needs 
cannot be realistically anticipated or to the extent that 
one-time-only reviews or p/obes are more likely to be 
needed, the organization of evaluation work should be left 
unstructur J until it is time to actually get a study under- 
way. ♦ 



The principal message of this chapter will be for 
people in charge of RELS to give some thought as to what 
evaluation responsibilities can be anticipated, but not to 
invest in evaluation resources until time to evaluate is at 
hand. 



But that, does not help a committee planning a Regional 
Environments Learning System and trying to prepare a 
proposal for funding. ^/omething needs to be said there 
about program evaluation. 



Ix»t us suppose, in the spirit of the third chapteH, 
that the staff of LTVEC, acting for a growing rjumber of 

include a .statem^it such asjt^, following : 
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EXCERPT FROM HYPOTHETICAL REUS PROPOSAL 



T 

PftQGRAM EVALUATION . The activities and services of the 
RELS ^X^ould be revieN^icd both formally and informally according to an 
Evaluation Plan. This) plan would be developed each year by staff 
and approved and supervised by a subcommittee of the Board of 
Directors. The Plan would include meeting all requirements for 
evaluation set by government i participating, and funding agencies. 

One staff member, a person having some formal training in 
program evaluation, (at' the time it is expected to be Associate 
Director Pauline Grubb) will have operational responsibility for eval- 
uation work. It is not anticipated that additional staffing for eval- 
uation will be made except as needed by particular studies. Those 
studies may be done by subcontract, by outside personnel working 
under the direction of staff, or by temporary or permanent staff 
members. 



This staff member will maintain a file of issues, program com- 
ponents, ventures, or indicator variables which might be evaluated, 
plus information to the resources available in the regon to assist in 
their evaluation. Frequent suggestions will be made to the staff or 
Board as to the possible utility of particular studies. 



One of the matters needing evaluation (perhaps to be done even 
before the proposed startup date) has to do with the "mental" identi- 
fication of the, RELS with state and federal Environmental Protection 
Agencies. Opinions of some 'support group leaders is that EPA stands 
in the minds of many citizens as idealistic, unrealistic and indifferent 
to the problems of unemployment and economic development. The 
depth of these feelings might determine how the RELS should ap- 
proach the public in its early efforts. A small number of structured 
interviews with a carefully selected sample is being considered. 



To audit the exerdse of evaluation responsibility within the 
RELS .it is expected that the Subcommittee mentioned above will retain 
an evaluation specialist (with no other involvement in the work of the 
RELS) to inquire at least once a year as to the activities and records 
for program evaluation, as a fee not to eKceed $200. 
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What $5,000 Will Buy . The best guess we have at 
this writing is that the optimum size of a RELS in a well 
populated region will reuire an annual budget of perhaps 
$250,000. Administrative costs should not exceed 20% of 
that and formal evaluation costs should seldom exceed 10% of 
that. It multiplies out to $5,000 per y«ar. It seems reason- 
able to talk about a hypothetical RELS that sets aside $5000 
a year for program evaluation' purposes . 



Five thousand dollars is a very large amount of money. 
But it does not buy much in this labor intensive world. 
Let us list several things that might cost five thousand 
dollars. 



Perhaps we should repeat that ^nly one of those small 
purchases is available for our estimated evaluation budget. 
Ob viously it is important to be selective. ^ 



But it is also important to acknowledge that there are 
no fixed fees in this line of work. One can sometimes get a 
better study for a thousand dollars than for twenty thou- 
sand. One important difference is whether or not one can 
draw on volunteer labor, people who have other reasons 
than money for helping out. People already on salary, 
retired people, spouses, and students are among those who 
often can be drawn into the evaluation work at low cost. 
They ysually will need coaching and coordination, but with 
careful arrangement, that can usually be obtained for 
reasonable sums. 



To hire people special]^ trained in evaluation work is 
likely to yield higher quality work, but if those people are 
expected to carry out the bulk of the evaluation activity, 
and it they charge appropriately for their time, they will 
quickly exhaust the available funding. 



Most evaluation specialists will advise the RELS people 

to commit themselves early to the evaluation information 

they will need so that time and opportunity arc available to 

m^ke the measurement. s . This makes a lot of sense. But it 

also often defies the reality of not knowing what information 

will be needed , what requirements will be most pressing , 

and what problems will be mbst vexing. If the evaluation 

.studies are to serve the needs of the people operating and 

benefitting from the RKLS, it is un^se to commit evaluation 
resources far in advance* 




(a) 
(b) 
(c) 
(d) 
(e) 



i the annual time a competent specialist 

a half-time graduate assistant 

a modest survey of the community 

a modest regression study 

a small case study of the program 
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Another problem . here is the turn-around time of 
research studies. It is regularly found that people very 
'much interested in evaluation information at the time the 
study is planned have quite different interests by the time 
the results come in. This problem cannot be diminished 
much b^ rescheduling the study or shortening its duration. 
What evaluation people study or shortening' its duration. 
What evaluation people have ^tQ do is to organizing the 
studies around those questioqfs that appear to be of rela- 
tively durable interest • V 

All this is fo say that the evaluation planning may bo 
[lone too far in advance and not far enough. It may rely 
too much on volunteer and amateur labor, and not enough. 
Since only a limited purchase is usually possible anyway, it 
seems best for the RELS plan to clearly indicate: where 
responsibility ^ for formal prc^pram evaluation will lie ; how 
assurance will be made that that j-esponsibility will be 
exercised; that there will be a regular review of the needs 
for evaluation studies; and that there are known resources 
outside the organization that can be drawn on for particular 
studies that may become needed- 

EVALUATION 'networks 



Part of the effectiveness of the strategy outlined above 
depends on a knowledge of the evaluation resources that 
exist in*' one^s own region and that can be tapped by tele- 
phone or maiL * ^ 



Paul Meadows,, a distinguished sociologist, visualized 
economic and social . development as testing a "delicate web 
of human accept^jice, " The development of an evaluation 
resaurce indeed tests such a web. We have found that 
most of l^he recent literature ot;^ "networks" is relevant to 
the development of resources to help with program eval- 
uation. 



Cohen and Lorentz , contribiJting to that literature, 
stressed that the planning and designing of people networksi 
was (in 1977) still "in its Infancy J' They saw such net-' 
worfts as voluntary associations in which a wide range of 
participants joined together out of a sense of enlightened 
self-interest (again comparable to the Creation of the RELS 
itself) • , 

Much of the literature on networks and networking is 
identified with the concept of "temporary systems" and the 
alleged existence of "invisible colleges,". Antecedents of 
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these terms are found in organizational theory and studies 
made by the research community. We visualize another type, 
of network, one which would be deliberately ppen to new 
people, to the addition of new resources. 



An outstanding "futurist," Hazel Henderson, described 
this form of network as having no headquarters, no formal 
leader, and no chains of command. It is also marked, she 
asserted, by free-form, self-organizing principles and is 
concerned with "issues, ideas, and knowledge." Henderson's 
analysis honors Thompson's call for retaining slack in the 
system. A word of caution should be spoken, though. 



Henderson argued that emerging networks have re- 
ceived scant attention from organizational theorists because 
they are too ephemeral, seemingly lacking substance or 
structure. We too are doubtful that networks tied closely to 
the ongoing operations of RELSs could be free-floating and 
dependable. We urge the RELS people to pre^)are themselve^ 
to . draw (from time to time) from existing networks of 
evaluation specialists, creating small temporary systems as 
needed to carry out the one-time-onlv evaluation work. 



Members of existing evaluation networks are to be 
found on any campus, in any large school district, in any 
state government. Finding dependable resources, of 
course, is more difficult. This is a major part of the 
responsibility of the preparatory work, identifying those 
resource people and^ centers that can be counted on for 
help. Personal discussions are needed, some traveling 
'around is to be expected, some careful study of their 
"portfolios" of work cqmpleted should occur. 



On the following page we have listed some people we 
have found jto be considerate and competent. They are not 
necessarily the best, but ortes we have. known and admired. 
They have agreed to discuss program evaluation ^nd to help 
inqui' -srs get in touch with other evaluation people. 



PROGRAM EVALUATION RESOURCE PERSONS ANf CENTERS 

/ 

The people named here have agreed to help Inqui/erS Identify materials 
and people for evaluating programs of envj ronrpental' education. 



Alabama 

Alaska 

Arizona 

Cal i fornia 

Cal i forn la 

Colorado 

Connect i cut 

D.C. 

Georg i a 

Hawa i i 

I II inols 

I II inoi s 

Indiana 

I ndiana 

Iowa 

Kansas 

Kentucky 

Louisiana 

Mai ne 

Maryland 

Massachusetts 

Massachusetts 

Michi gan 

Mi ssour i 

Missour^i 

Minnesota 

Minnesota 

Montana 

Nebraska 

New Hampsh i re 

New Jersey 

New Mexico 

New York 

New York 

New York 

North Carol ina 

North Dakota 

Ohio 

Ohio 

Ok I ahoma 
Oregon 

Pennsy I van i a 

Pf^nnsy 1 van !a 

Rhode Island 

South Dakota 

Tennessee 

Texas 

Texas 

Utah 

Vermont 

Washington ^ 

West VI rg in ia 

cons i n 
Wlscqns in 



Theodore Pinnock, Motton Hall, Tuskeg<^e Institute, Tuskegee 36088 

Terry Elofson, USA Juneau Ave, Fort Richardson 99505 ^ 

Len Cahen, Assoc Dean of Ed, Arizona St U, Tempe 8528I 

Center for the Study of Eva! (Adrianne Bank), \kS Moor6 Hall, UCLA 90012'* 
Far West Lab (Paul Hood),. 1855 Folsom St, San Francisco 9'tl03 
Mary Lee Smith, IOO6 6th St, Boulder 80302 
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Evaluation Technical Assistance ^ Since project funds 
should be used primarily for program operitioilte and o^ly in 
a small way for evaluation, how can RELS people make the 
most of the technical assistance that is available? It is 
important to realize that some agencies have a history of 
providing technic^ assistance of one kind or another, -The 
funding agency should be asked- Inquiries should be made 
o{ state departments of education and universities- Many of 
them support evaluation centers that provide free or low 
cost assistance to agencies in their region- 



Usually the technical assistance is available for con- 
sultation but not to do evaluation work- Often technical 
consultants will recommend personnel as well as methods ..for 
accomplishing the work- These recommendations should be 
examined carefully- If possible, other specialists and 
various interested parties should be consulted- The steps 
outlined in^ the previous chapter for writing contract 
should; be considered, even if the decision is made to do 
the work with "inside" people - 

To the extent thai: RELSs are the quintessence of 
informatioji systems, they will have a special opportunity to 
draw upon the insight and conscientiousness of people for 
program evaluation pXirposes- 



What can RELS project directors challenged to produce 
evidence of program effectiveness expect to receive from 
the evaluation community? Something helpful, but often not 
enough- No evaluation group or plan of action can prove 
that even a very good project is having a substantial im- 
pact- Evaluators can provide better information, a new 
perception of responsibility, and some political re- 
spectability.. 



Sometimes these benefits will come more quickly if a 
formal evaluation unit, even a small one, is set up in ad- 
vance- Certainly the RELS collective inquiry routines are 
more likely to be used effectively this way- However, the 
unit will be less able to deal with new ^valuation respon- 
sibilities- The creation of an "evaluafion^ responsibility ^ " 
something less formal than an evaluation unit," may more 
slowly allocate the resources — to greater advantage- 

/ ■ 
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An ongoing system for monitoring program processes 
and impacts can draw from^ many different technical com- 
munities. Their different methods will not minimize th/s 
expenditure of funds or lead to entirely compatible con- 
ceptualiza(ti6ns and val^iings, but the diverse school -and 
community goals are more likely to be respected. Such an 
approach also responds to Thompson's call for maximum- 
flexibility within the context of a changing world. 
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The previous pages are intended to 
provide a few examples, explanations, 
descriptions, and teachings related to 
evaluation, and particularly 'to the evalua- 
tion of environmental educational pro- 
grams. The field of evaluation is broad, 
and coverage in a single document ie 
necessarily superficial. 



For those who seek a more serious 
and extensive engagement with the litera- 
ture of evaluation , or those who want 
only to explore a single topic in greater 
depth, the following two pages ^ave been 
included. The Bibliography provides 
references to further readings; the 
Topics and Authors page gives additional 
entry points into readings by key persons 
in the field of evaluation. 



An ultimate hope would be that the 
network of experts and practitioners 
within the field of environmental education 
might come to overlap in meaningful ways 
the network of significant persons in the 
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